/r/talesfromtechsupport

Photograph via //r/talesfromtechsupport

Welcome to Tales From Tech Support, the subreddit where we post stories about helping someone with a tech issue.

> FILTER POSTS BY LENGTH <

All Short Medium Long Epic None

> WELCOME TO TFTS! <

Hi, Everybody!

TFTS is where we post our amazing Tales From Tech Support, including but not limited to:

  • Incredible Feats of Networking Heroics;

  • Tech Troubleshooting Under the Direst of Circumstances;

  • Unsolvable Problems Cracked by Sheer Genius and/or Pure Luck;

  • Moral Support after Having Dealt with Difficult Clients;

  • And of course, Stupid User Stories!

There's a bit of a lull in the queue just now, so kick back, grab a cold one from the secret tech fridge behind the server rack, and share your best tales among friends here at TFTS!

> TELL US YOUR TALE! <

Rule Ø : Your post must be a written story about tech support. Please do your best to make it interesting, readable and concise. One post per 24 hrs please.

Rule ∞ : Don't be a jerk. Just don't. We get enough jerks at work. If you can't participate without being one, we don't want you here.

In particular, Reddit Truth Police will be banned without warning. That will happen.

> MODERATION INFO <

  • Posts and comments that do not meet these guidelines may be moderated.

  • Breaking Rule 3 or Rule 7 may result in an instant ban from the sub.

  • Visit the TFTS Wiki Page for more details about our moderation policies.

  • Please feel free to message the moderators with any questions.

  • DO NOT USE CHAT OR PM.

> COPYRIGHT POLICY <

> SPAM FILTER ALERT <

If your post does not appear, or vanishes after posting, it may have been flagged by Ye Olde Spamme Fyltre.

Do not repost the same thing again, that just makes it angrier.

Send the mods a quick message instead.

You may also review our lists of Things That Aren't Tech Support and Officially Retired Topics, just in case it wasn't the spam filter after all...

> SISTER SUBREDDITS <

> TALE SUBREDDITS <

> THE BEST OF TFTS <

> TFTS ESSENTIAL LINKS <

> BYE EVERYBODY! <

TFTS Top Tales - The Complete TFTS Archives

/r/talesfromtechsupport

771,806 Subscribers

454

Admin Rights and Wrongs

My company recently upgraded from Windows 10 to 11 and one of the biggest issues are some of the files on the network drive went missing. They are easy enough to restore, but they involve signing into the computer as an admin and disable offline files. I just had a call a early today that I wanted to share.

Me: Thank you for calling the IT help desk this is 'MY name', How may I assist you?
Customer: Yes, I recently upgraded to, You know what, it doesn't matter what happened. My files are missing, I need you to restore them.
Me: "Do you mean the windows update, If so this has been a problem with the upgrade itself. Do you mind If I sign into your computer, there is something I need to run first."
Customer: "What do you need to fix my computer. Are you saying I need to call IT every time I have this issue?"
Me: "Ma'am I will need to enter my admin password to fix this issue, If issue does occur afterwards then we can send this over to another department for a more permanent solution. "
Customer "So hat you're saying is that you're not going to be able to fix my issue"
Me: "No ma'am that's not what I am saying at all, yes you will need to call the IT help desk if this issue does occur, since only a system admin can fix. Now do you mind if I sign into your computer."
Customer "Fine, but I want a guarantee this issue will never occur, again."
Me "Ma'am I can't do that. There is never a guarantee that the issue won't reoccur"
Customer "Fine sign in, but I want it escalated regardless if you fix it or not. I'm a very busy woman, and I can't call the IT help desk for every issue. "
Me "OK I'll escalate, Now if you could give me the computer number and save and close any confidential documents that might be open, I should be able to assist you. "
Customer Shouting " What do you mean close my documents, you;re not goign to to delete anything are you?"
Me"No ma'am, I just need to run some processes on the computer and I don't want to sign in to a file that you don't want me to see."
Customer" I don't have any files open, and If I did I wouldn't want you to see them"
Me "OK that's what I asking for."
After that I sign into the computer, The customer is mostly silent, but under her breath I hear her muttering how useless IT is. I was able to fix part of her issue, but and sent it over.

60 Comments
2024/11/25
15:14 UTC

448

Monitor-eating office cat

Small office customer. Sold a new, bezel-less 24" LCD monitor to said customer. Customer takes it on-site and hooks it up, all works well. It sits on a counter by a large window.

Customer calls me a few days later. Says the new monitor is dead. The power light is on, but no signal. I have him try a different cable, different port on the PC, etc., but nothing. So we process a replacement, swap it out, and chalk it up to being DOA.

Customer calls me again a few days later still. Says you're not going to believe this, but this second new monitor is doing the same thing as the first. At this point I'm thinking their office must have a power problem or something that's killing monitors. But I decide I'll take a truck roll and see for myself.

I go on-site and bring a third new monitor with me just in case. I open the door and see a very pretty cat walking on the floor. I look at the old, original monitor which was replaced by the new 24". It's an old 17" LCD from a decade ago and had thick, beefy bezels as monitors from that era did. I see some bite marks at the top corners, but they're just on the bezel so no actual screen damage.

It's beginning to add up. Their office cat had been chewing the corners of the old thick-bezel'ed monitor. Which was fine, until they got a new monitor that had no bezels at all, and all it took was one bite from the cat to pierce the LCD itself. Twice. Once they were made aware, it was easy to see the teeth mark in the corner of the LCD of the new monitor.

Customer ended up getting a used 22" monitor with thick bezels. Cat still chews the corners.

EDIT: I found the pics!

https://ibb.co/b3s84DJ

https://ibb.co/Vx2H1sX

https://ibb.co/8rVKd56

26 Comments
2024/11/20
19:46 UTC

964

Yelling at IT staff does not a business continuity plan make.

This is from a few years ago. I was working at a medium sized company as an IT sys admin. The company had just recently moved to a new location that was able to more comfortable accommodate its operation. It had an on-site call center as well as a medium-scale manufacturing/repair center. Since we were new tenants and everyone was now under one roof, many things were still being figured out.

One day, we got notice of a gas leak in the manufacturing area. We didn't have an alarm system for a gas leak so people were running around telling everyone else that there was a mandatory evacuation of the building. The IT people all had laptops so we all grabbed them and made our way to our cars. By coincidence our director of IT and the head of IT support were on a business trip. As I'm walking out the door the Call Center Director (I'll call him Cal) start yelling at me and the other Sys Admin. "Hey, what are you guys going to do!?"

"Go to our cars."

"No, no you can't. We can't receive calls. You have to do something!"

I turned to my coworker and we both realized that the call center still used desktop computers and soft phones. They couldn't do their job. Cal was red in the face trying to slowly let people out the door to the outside. It was then that the fire department arrived probably to clear out the building officially. So I asked Cal, "What's your plan if there's a fire? Just do that."

"What? No, you need to do something."

I shrugged. "We can't do anything. The phone system probably doesn't work off of VPN." I was guessing at that. "Just follow your plan if there's a fire."

"You guys never gave us a plan for a fire." Cal responded.

Because of course it's IT's job to develop a business continuity plan for the entire company. More people were streaming out. It was then I decided to ignore him and go to my car. I tried to call the Director of IT in the slim chance the airplane diverted or was delayed. No answer. I looked up in the company SharePoint site for a business continuity plan or fire plan or something. But only found stuff for IT, including our offsite backup servers and how to run IT operations from VPN. There was nothing about moving our softphones to/through VPN.

Cal knocked on my car window after everyone was out of the building. "Well?!?"

I explained that there was no business continuity plan in the SharePoint site and IT didn't have anything in place to shift the softphones to VPN. Plus we didn't have enough laptops to support even half the call center. Cal didn't like my answer and walked over to the CEO who was the fire department. I could see Cal pointing at me and yelling. Clearly we were losing business. And clearly it wasn't just IT's fault, it was mine and mine alone.

The fire department cleared us to go back in after about 45 minutes. Later that day I had two meetings with Cal and the COO scheduled. Since IT was missing both leadership positions to travel I was the scapegoat. The first meeting was cancelled and the second the CEO stepped in and cancelled it since it was really the job of the Director of IT and a lowly sys admin shouldn't be in these meetings.

Nothing bad happened to me when the IT Director returned. And the company hired a consultant to develop an actual business continuity plan for fires, weather and other events. Turned out, IT shouldn't have a button they could press in the event of a gas leak. For several months Cal scowled at me after that every time we passed in the hall.

TL;DR Call Center Director assumes that because his department uses computers, any problem becomes an IT problem.

126 Comments
2024/11/20
17:39 UTC

718

When DNS is just a dude

Hey! I've been lurking through this subreddit for more than a decade at this point; I have now become a telecom engineer, and I have some stories to give back to this wonderful place: this is the story of when our nameserver was just a dude.

I had just started working a volunteer position for a local NGO, I was already studying engineering and had been working with these guys for a while, and as the resident young guy that works with computers in a place filled with old people, I just slowly drifted into an IT of sorts; after getting Office running on a couple of laptops and fixing and documenting their heinous email situation, I got some one-on-one time with our librarian:

$librarian: Hey u/benjazio_xd, can you help me with something here?

There was a reference collection of books for internal use, around 30k books in total, managed by this one guy who also cared for the NGOs extensive paper archives, which were around a hundred years old. He's a cool guy who actually turned into a great sidekick for many projects I did while working there, and we remain friends even after I left.

$librarian: You see, we've had this ILS for a while, and I've been told it has an open access catalog so our guys can see what we have and come pick it up, but I've never gotten it to work right, could you take a look at it?

An Integrated Library System (ILS) is a piece of software that tracks pretty much everything inside a modern library: inventory, loans, labeling, shelving, late fees, you name it. They are very niche software but also extremely powerful: they are the beating heart of many libraries, big and small. This one was hosted on a local server in the office itself, and a quick browser check to the local IP address of the server revealed that it did, in fact, have an open access catalog.

$librarian: It's supposed to be on our webpage, but I've got no idea how it works and no one really explained it to me when I got here.

Sure enough, there was a link on our webpage that just returned a blank page every time, and not only that, it seemed to be an internal URL on our webserver, which didn't really make sense considering it was on a different machine halfway around the world.

Nothing in the world would have prepared me for what I saw when I clicked on "Inspect".

$me: So, um, has anyone ever told you anything about this before?

$librarian: the previous girl that had my job told me that the page had to be updated every couple of weeks, and left me a couple of links I had to follow, but she never taught me how to do it and that was like five years ago.

Jesus Christ, this hadn't been working in a long time.

In this blank page was actually an iframe, which pointed to the frontend on our public IP address. This was janky and unnecessary, but what turned it into depravity was one key little detail: we had a dynamic public IP address.

This meant that for years, someone had to connect via FTP to the site every couple of weeks, go to this page, and modify the iframe so the IP address matched to the current one we had. There were no notifications set up either, which meant someone had to notice and tell the librarian that this was going on in order for this to even work, and when they changed librarians no one bothered to write this down, and so that site was just permanently broken: Our dynamic DNS solution was just having a dude update a file on a remote server whenever they noticed the god damn page was down.

$me: This is extremely stupid, how did nobody notice this earlier?

$librarian: you're the first guy here who actually knows enough to care.

My heart sank a little. Apparently this guy had been complaining about this for years but because the dev team for that website was long gone no one had bothered to get someone to look at it. It was an unfortunately common scene in this place, and it was the reason that made me leave it some years later.

$me: Right, this is going to take me about an hour to get everything set up, but I'll get it fixed and running before the day is done.

My solution was just to get a DDNS provider and hook it up to a subdomain of our main site using a CNAME record and just changing the link to the page to the new address. This was fairly low traffic website and just have the server directly respond to requests was fine. I used a small script on the local server running every five minutes to update the IP address to the DDNS provider and that was it: it now just worked on its own.

$librarian: You have no idea how much rage you've removed from my system, let me buy you lunch tomorrow.

We got a static IP a few months later, and I made a friend in the process.

21 Comments
2024/11/20
04:50 UTC

1,134

The Customer Who Didn't Understand 'Turning It Off and On Again

I work in tech support for a fairly large company, and I’ve had my fair share of bizarre calls. But this one really stuck with me.

A customer calls in, and the first thing I notice is that they’re clearly frustrated. I ask for details, and they explain that their computer is “just frozen” and nothing is working.

I tell them, as calmly as possible, “No worries, let’s start by rebooting the computer. Please hold the power button for 10 seconds to turn it off, and then turn it back on.”

There’s a pause on the line, then: Customer: “I don’t know how to do that.” Me: “You don’t know how to turn off your computer?” Customer: “No, I don’t know where the power button is.”

I’m trying to stay professional at this point, so I walk them through it. I even ask them if they can find the power button on the actual device. They respond that they don’t see one.

So, I ask, “Can you look on the side or the back of the computer for a button or a logo?” Customer: “It doesn’t have one.”

At this point, I’m a little confused, but I decide to walk them through the process anyway. I start asking if they see any lights on the device. They tell me no, nothing is lighting up.

Then it hits me. I ask, “Are you sure you're working with a computer?” Customer: “Well… no, I’m looking at my microwave.”

This person had been trying to reboot a microwave for 30 minutes, thinking it was their computer. After a long, awkward silence, I confirmed that microwaves don’t have the same functionality as computers, and recommended they try restarting their actual computer instead. They were extremely apologetic, and I just couldn’t stop laughing after I hung up.

Never a dull moment in tech support, folks. Stay strong out there!

141 Comments
2024/11/14
21:59 UTC

393

8,300 lines of dependent code to *drum roll* create one record

**For context - this is a project I was called in to as an emergency resource. Basically, come fix our mistakes...anyways...

Oh yeah. Just found this one recently. One transaction runs 170 lines of code to verifying the creation of a single record across 3-4 classes.

The total line count of those classes and their corresponding test class is just over 8,300 lines of code. One of the classes on its own is over 2400. I saw a method that had over 300 lines just on its own. What a trooper.

The first kicker (maybe not, there's a lot of great stuff in here....8 layer if else statements....)

Okay, one of my favorites is that the big mama class is a global class and is integrated with an external system. This system, as evident by dozens of unnecessary permission checks, can have multiple users hitting this class at the same time.

That's a real problem because -every- -single- -method- is static and **THE SOURCE RECORDS ARE STORED AS GLOBAL STATIC VARIABLES**

For those who may be unaware or just unfamiliar, "static" means that there cannot be multiple instances of that variable even if multiple "things" are referencing that method/variable. In other words, if you have two requests come in at the same time - you could very easily end up swapping the source records mid process and corrupting the data on one or both sets of data. And you would never know because, while corrupted, all the data is likely still in a valid format and will pass through the system without a system error. Kind of a big deal when it's directly associated to every deal in your sales process.

But the thing that really prompted me to post this was such a level of "I don't care about the next guy" that I am actually stunned. I think I may be in denial still...

There is a class, and he's amazing. We know the rules - don't hard code values that are susceptible to change. If at all possible (and reasonable) don't hard code a constant that may change in the future.

This class, I shit you not, is just shy of 500 lines of grade-a, organic, non-GMO constants baby. They're global, they're hard coded, and they're susceptible to change.

But, that wasn't enough. It's the gift that keeps on giving. After spending the better part of 12 hrs wishing I was dragging my head across the pavement at high speeds, I noticed the comment at the top of the code (slightly paraphrased).

"Class to hold the static values...to avoid hard coding."

Sometimes, man, I really do wonder. I feel like I have done a pretty good job remaining positive about this absolute mess of a transaction - but why they gotta spit in my face like that? It's hard coded, for 500 consecutive lines, right below that message. That's EXCLUSIVELY what this class is.

C'mon man. Just....DAMN!

**Final bit of context, because idk what everyone's familiar with, this platform has built in functionality for getting the exact values at the time of running. There isn't a single thing in that constants class that needs to be declared as a hardset constant in the code. And there are no checks to verify that the transactionId remains consistent.

64 Comments
2024/11/13
05:42 UTC

928

The program changed the data!

Years ago, I did programming and support for a system that had a lot of interconnected data. Users were constantly fat-fingering changes, so we put in auditing routines for key tables.

User: it (the software) changed this data from XXX to YYY…the reports are all wrong now! Me: (Looking at audit tables) actually, YOU changed that data from XXX to YYY, on THIS screen, on YOUR desktop PC, using YOUR userID, yesterday at 10:14am, then you ran the report yourself at 10:22am. See…here’s the audit trail…. And just so we’re clear, the software doesn’t change the data. YOU change the data, and MY software tracks your changes.

Those audit routines saved us a lot of grief, like the time a senior analyst in the user group deleted and updated thousands of rows of account data, at the same time his manager was telling everyone to run their monthly reports. We tracked back to prove our software did exactly what it was supposed to do, whether there was data there or not. And the reports the analysts were supposed to pull, to check their work? Not one of them ran the reports…oh, yeah, we tracked that, too!

72 Comments
2024/11/12
01:53 UTC

412

The Halloween Spider Attack

So it's Halloween and as usual everyone is in the office (building B) for our Halloween party. On top of that it's time for the regular Tabletop/IRP review. So everyone in the department from desktop support to security, Property management, the CIO and CISO are all crammed into the hot conference room. Luckily I'm not as I asked to work from Building A that day

Anyway, we started going through our IRP scenarios. With some members strategically barred from answering. Our new security analyst pipes up and says we have spiders attacking our network. Confusion follows, is this a scenario? Then slight panic, this isn't a scenario. CISO ask where are you seeing this. Analyst says, we just got a critical ticket. Someone opens the ticket and reads web spider attacking printer in building A This makes no sense so network, security and webdev start checking their various metrics and logs. I'm in building A so I go to check the printers.I find spiders all right, tiny plastic spiders all over the flatbed in the exec suite.

TLDR: someone messed with the printers, a game of telephone leads to a VIP (automatically critical) ticket saying our printers are under attack. Turns out someone just covered the printers in plastic spiders.

16 Comments
2024/11/11
01:41 UTC

548

The time my dispatcher thought we were experiencing a terrorist attack

A bonus story for today but going to be vague. won't get in trouble if im not but still probably a good thing if im vague.

So after some of the terrorist attacks in the early 2000, my company contracted a big aerospace company to build a system to automatically detect bio weapons. Said system is very expensive and requires a lot of maintenance and has multiple people monitoring it remotely.

One day my dispatcher received a call from a remote monitoring site saying that we need to check one of our machines because it's retesting a sample. My poor dispatcher interpreted this that it has detected something and being the only one on my shift trained for that system they called my cellphone directly. Dispatch doesn't know our cells because we have radios. So they got management to call me because no one wanted to talk about this over the radio. Was given direct orders to tell no one and go immediately to the machine. I arrive at the machine with everything running fine with no fault lights. So I logged in. The issue, a bad test tube. This machine has multiple so if it fails to get a "false" but also fails to get "positive" it will retest with a new tub. Nothing major, it just runs another test, the test do take awhile though so i got to sit and watch a screen. Did call dispatch back though and told them we are not under attack so they can calm down.was very over dramatic.

32 Comments
2024/11/08
15:01 UTC

149

The disappear fault

Time for a couple more badly written stories, words are hard and I never went to college. You get what you get.

Get some doordash, maybe some Adderall or whatever your vise is and enjoy

My job is tech support related but not directly. I work on anything from servers, networking, to automation(belts, motors, bearing) and PLCs. I'm a jack of all trades and definitely a master of none.

:The disappearing fault

So one day Operations calls us due to an output module fault. It looks like 7 modules lost communication. Well we first check the com cables, 2 40pin cables that create a loop for 4-19 modules. They seemed fine but Admittedly avoid these cables because I hate them, bulky, bad retention mechanism, and likely to have more problems just from touching them. All the cables for controlling coms, gates, actuators, and safety loop go to a backplane that slots into the main control PCB. So we replaced the main PCB, nothing happens except for even more faults. Then we got a second one, kinda worked just different faults this time. So we got a third one, most faults gone except one but communication is back for everything. At this point I called a remote SME, system matter expert. Who says to swap the board with another module to see if the fault follows the main board or stays with the module. One problem, it does nether, it just disappeared. Doesn't make sense to me but it's gone and the machine works.

Main lesson learned, just expect all your parts are bads.

:When the SME is wrong

So Operations calls about a machine intermittently stopping for a safety loop fault that never calls out where the fault is in the machine. The machine will act good when not processing but after 30s to 5m it will fault out. We arrive and start looking at interlocks but couldn't find anything. We keep pressing start till we get the fault to show up and not immediately disappear. We checked a 24v safety aux contact attached to a relay, even though it didn't test bad it's so common we replaced it. After checking every interlock we can't figure it out. So we call the remote SME. One piece of info I did have to diagnose the issue is the aux contact has no power. Letting the SME know this he first says to replace it, I told him I already did, also told him I'm not trained on the equipment though so I'm at a lost and need schmatics. He emailed me the schematics and also wanted me to follow the schematics after the aux contact which didn't make sense to me because it wasn't getting power. I felt I should trace where the power comes from and see where I get it back. So I lied to him that I would then with the self confidence of a stupid person, I did my own thing. Found power going into the safety loop at the breakers, the breakers have the ability to tell the computer if they're tripped, but not coming out. Started shaking the connectors on the back of each one until I found one that would make the machine go ready and not ready just from wiggling it, yes I wore gloves if anyone from OSHA is asking. After the machine was down for 5 hours, one aux contact and one breaker later it was fixed.

To explain what was happening, the machine vibrates when running due to motors, bearing and belts. This vibration would cause the tab inside the breaker to disconnect momentarily causing the machine to stop due to the safety loop opening.

:when the senior tech doesn't compute

So on day shift they had a machine go down due to the output modules not communicating. The senior tech(for day shift) found the module not communicating and replaced the board but still wasn't able to fix it. Even shift with the 2 most senior techs out of everyone refused to touch it. Finally when i came in on night shift and was questioning why we had a machine down i decided i would look at. Still not trained but from knowledege of last time I asked if anyone configured the board. All evidence pointed to no. So one call to the SME for the document on dip swich configuration and crawling inside the machine later the machine worked...

Unsure if anyone else shares my frustration but fixing stuff more experienced and trained people shouldv'e give mangement unreasonable expectations of your ablities. I love solving problem, i don't love being put on a pedistol.

Btw the down time of that machine probably cost $150k-300k

:how to solve a random persons problems from 500 miles away.

So the techs at my company have a facebook group for memes but also for help when SMEs are no help.

A person in another state posted they have had a machine down for over 7 days. The machine would only fault out if you tried to run it. With the fault being a communication fault from the operator PC to the on site OCR server, Optical Character Recognition. The issue was they could ping the server, and PC and server would show connected in their respective software. They even ran a new cable from the switch to no avail. I guess no one on site or the SME thought to actually see what the switch was reporting. I had access to see the monitoring of every facility just not make switch configurations. I was bored and looked them up and saw a ton of errors. The port was configured correctly, so most likely bad port.

So I messaged the guy. We got me =me, tech= guy from that facility, and supervisor = his boss

Me- hey, i saw your FB post i think it's the switch port

Tech- we are going to reboot again

Tech- I'm going to make a group chat with my supervisor

---new chat---

Me- hi, i think this is an issue with the IDF switch, do you guys have anyone with cisco CLI training and log in.

Supervisor- i think so but he hasn't logged in awhile

Supervisor- SME says to check switch at machine, we replaced it but that didn't work. SME now says to replace IDF switch

Me- before stopping all operations lets just try another port

Tech- we need a ladder

Me- i see the switch lost power recently, did you guys have a power outage.

Supervisor- actually yes, thats when the problems started

Me- please take the cable from port 6 and plug it into port 8

---Note, port 6 is for the machine having problems, port 8 is for a machine that is working

---23 messages and 4 hours later of being ignored

Me- please take the cable from port 6 and plug it into port 8

Supervisor- that worked we think port 6 is bad

Me- plug the cable ftom port 8 into port 6 and see if it faults out too

Supervisor- it does

Me- that comfirms 6 is bad, have your tech open and cofigure another port and label port 6 as bad.

Supervisor- thank you!


Moral of the story sometimes you need to repeat yourself i guess. Still working on being assertive.

On the plus side this interaction helped me pass the interview to become a SME, just waiting for an open postion.

:the normal tech support call

So us machine techs are only supposed to fix anything related to machinery and their functions "processing infrastructure side". We consider anything not related "Lan side"(printers and supervisor computers).

One problem, one onsite "Lan side" tech covered like 6 plants almost all 120 miles apart. They could drive 5 hours for one call and responce time is like 2 weeks.

Due to how over streched this guy was, even though he didn't want my help, and my interest in tech I would help when i can. It was against the union contract but keeping the bosses happy was in everyones interest. I mainly would just help with printer problems and was well known by management for solving printer problems. After the print server/directory failed i was the only one to get the printers working while we waited for it to get fixed. Anyways here's the story.

-Over the dispatch radio

Supervisor- hey OP can you help me with the printer by machine 9

Me- On my way

---i arrive stage left

Supervisor- I can't get the printer to print, i think it's broken

Me- please bring up what you're trying to print

Me- press print or ctrl p please

Me- can you select the printer labeled "printer by machine 9" please instead of "print to pdf"

--- exit stage right as it starts printing ---

When i was asked to work on their networking though i would say, "only if you can provide a network diagram/topology" . I perfectly well knew they couldn't because they never made one for their side of the network. Their network closet was an actual birds nest. Like you had to walk on the cables to get to the rack, like the rack looked like vines covering a tree and all the walls. There was more un used cables ran in there then used ones. Patch panels, what patch panels. Idk how it looked like that for only having lile 8 switches, 2 firewalls, and 2 routers.

Grammerly broke like half way through this so sorry not sorry.

19 Comments
2024/11/08
11:26 UTC

412

Santa Flaws

In days gone by there was a project to refresh our site.

Pallets upon pallets of kit arrived. Desktops, monitors, as many as our storeroom could take. Desks were stripped and rebuilt. So shiny. So new. So uniform. So factory default. It had a certain unblemished beauty akin to thick, level, settled snow.

"Are we going to refresh the network cabling too?" I ask, thinking of the decades old CAT5 poking out through holes run directly from the server room, not a single RJ45 wall socket in sight. The response I receive only adds to the Christmas card like image in my head. The absolute crisp, delicate serenity of acoustically insulated silence.

Santa visits often, and is generous. Boxes arrive, boxes are opened, boxes are sorted for recycling.

Some of the boxes are different to the others. Some are flatter. Some are heavier. These boxes arrive, these boxes are not opened, these boxes are set aside in the corner.

Santa has been at it quite a while now. We must be the goodest and bestest boys and girls in the world. Boxes arrive, boxes are opened, boxes are sorted for recycling.

"Is Santa going to keep coming all year?" I ask, aware of the quantity we're accumulating. Such exquisite, pristine silence. Boxes arrive, boxes are no longer opened, boxes fill up our storeroom.

PCs are ready to be imaged. So exciting! All the new toys, so new and so perfect, so delightfully fresh and identical. So full of baseline, known condition, uncompromised promise.

Images deploy, oh no. Such strange and erratic behavour! Intermittent faults and discrepancies! If only we had gone for the network cabling refresh. What a sad little disappointment on the happiest day of the year.

I think all is not well with Santa either. Boxes arrive, boxes are shunted straight into stores, boxes pile high. We are given the schedule for several more weeks of deliveries.

"What are we going to do with all this equipment?" I ask, concerned. The snow is beginning to turn a little yellow, yet still there is silence.

Maybe something is wrong with Santa. I write Santa a short letter. Santa replies that this is none of my business and to mind my own.

I am sad that Santa would speak to me this way when up to now he has been nothing but generous. I write Santa a longer letter this time. One with a count of the total number of desks on site, the number already set up, and the number of kits in storage compared to the number of empty desks.

Santa calls me immediately. "Err.. are these numbers right?". Santa and I become close friends, he calls me every week now. He only wants to talk numbers and dates for deliveries though. For many many weeks we only talk about the same numbers again and again, and I tell him he has been far, far too kind. Surely there must be other very good children in the world as well? I begin to suspect Santa only wants to talk to me so he can tell all the other santas who to speak to if they find out their generosity was misguided.

One of the other santas writes a letter to me, Santa2 is very worried he has missed us off his list. Santa2 wants to know if we received an extra extra special box. I tell him we have that very special box yes.

"What about the other boxes?" I ask, wondering about the tower of switches in the very far corner of the storeroom. No one has said we can open those yet. Silent Night.

I count again the number of desks in the entire building and divide by 48. On no, Santa. We have been sent far too many of these boxes. There must be several good boys and girls with their own sites that didn't get their switches by mistake! Several! These can't possibly be all for us?

I'm worried about Santa2 now. I write a letter to Santa and Santa2 about the switches. "Santa was on a temporary contract and doesn't santa for us anymore". Oh. Santa did not even say goodbye.

Santa2 did not say anything about the switches. This time I write a letter to six or seven santas. "Why do we have X times 48 switch ports for a site with Y number of endpoints?"

Santa2 calls me straight away. "..ummm. Can you send me a photo of them?"

24 Comments
2024/11/04
17:22 UTC

324

One disgustingly hot cup of coffee

And give me some disgustingly hot coffee. Black as midnight on a moonless night."

Blurp?” the nameless thing behind the counter said.

"Oh, pretty damn black, I should think."

Bloop.

"What?"

"Whiiiir.

"An overcast moonless night?"

Phssssh?

"I'm sorry? What did you say?"

Beep-blip boop.

"A moonless night, that is as black as coffee.

"Glurgle drip-drip-drip."

"Thank you."

 

As I reach for my fresh cup of caffeinated jetfuel a colleague pokes his head in and calls for my attention. Something is amiss in the digital dreamlands and I am needed to risk my sanity diving the datastreams once again. Two hours later and the binary gremlins have been placated once more, vanquished but not dead, merely dreaming in their deep slumber until some other unfortunate soul trespasses on their domain once more.

I finally sip my coffee. 

It’s not hot anymore. 

Only disgusting.

TL;DR: Dropped into flow mindspace solving a bug ticket and even forgot about my newly-made coffee.

45 Comments
2024/11/04
12:57 UTC

196

Pagers still exist? Not only that, there's still tech support for their central hubs, so to speak.

So, this isn't about a computer or an internet system, but I've posted "tech" tales beyond that realm that both remained and got more upvotes than down. That said, if you know a better place for this, drop me a line.

I worked for a communications company, that we’ll call Company A. We were on contract with Company B, to maintain their paging system.

Although pagers have been phased out in most places, some hospitals are an exception. This has one chief advantage that it can still work in parts of the building where you might not get a cell phone signal, provided the system is designed properly. Another thing is that it would be almost impossible for any type of spam or scam to get through, and very easy for a (company A) or (company B) technician to block if that ever did happen. In short, that means that it’s really only going to beep/vibrate for things that are work related.

Most of the transmitter racks were on the hospital sites themselves, connected to the backup power systems that held up the hospital’s critical stuff.

Back in the early days of cell phones, these paging systems once had the advantage of being more reliable as well. Unfortunately, (company B) never upgraded to a newer system, relying on 20+ year old technology to still be able to run 24/7. Mind you, the paging system was actively used multiple times a day for routine things, so this wasn’t something that just sits on standby until needed for an emergency.

The system was modular, somewhat like a desktop computer, and designed with field service in mind. Almost anything but the “backplane” board itself and the wiring could be serviced from the front of the rack. That, and the antenna would be on the roof of the building that the system was in. You could usually just unscrew things from the front, at most needing a Philips screwdriver if that (the original screws were thumb screws, but sometimes those got lost and you only had “regular” ones on hand at that particular moment)

The system was made up of 2 different power supplies, the motherboard, the keypad/display module, the amplifier module, and the “forget-me-not” module, whose purpose I forget, so we’ll just call it Module F if I refer to it again. Unlike most systems, the motherboard actually wasn’t the hardest thing to replace despite the necessity of everything connecting to it. The backplane board allowed you to swap out everything else, and it had nothing but connectors on it, specifically so it would be the least likely “module” to have anything go bad, BECAUSE the engineers realized it would be the hardest board to replace if it did go bad.

Both power supplies had the same voltages and current capacities, but one powered the amplifier, the other powered everything else. There were 2 indicator lights on the power supplies, one to indicate that it’s getting input power and it’s switched on, the other would indicate if it detected a fault with itself. Though the latter wasn’t foolproof (i.e. it could be “bad” without the fault light coming on) it DID help when it worked. Since they were the same PSU, you could swap them to see if the symptoms changed.

Most things that required more tech knowledge beyond basic computer servicing involved the antenna or configuring the software with a laptop. Even though it used a serial port, a proper USB adapter and their software design made it still usable with modern laptops, so at least we didn’t have to try and keep 20+ year old laptops working.

The most common things to go bad in the early days were the amplifier, and the power supply that ran it. That, and problems with the antenna wire connections, usually the ones on the roof that were exposed to the weather and such. Those usually resulted in missed pages only in the outskirts of the service area, assuming a preliminary “maintenance alert” didn’t show up first. Usually a total failure was either the motherboard or the “other” power supply that ran everything else. Module F rarely went bad until much later, and even the motherboard was pretty reliable until things started getting really old and rickety.

Unfortunately, that’s when I got hired on. By then, we were a frequent flier to nearly every location that still had these things. But none of the hospitals nor anyone else Company B supported had upgraded to a newer system. The only reason I can think of, was maybe that a new system wouldn’t be compatible with the old pagers and so they’d have to replace all of those?

Anyway, I’m dispatched for my first solo repair trip to Hospital 400. This site had totally gone down with no warning, which sounded like a motherboard or PSU issue. This hospital didn’t have an ER per say, so at least a missed page was less likely to spell disaster here. I just packed spares for everything we had, an antenna power meter, the laptop with the serial adapter and it’s charger, the whole 9 yards. We also had a little stack of paper explaining the command line interface and what you had to type to do certain things or reconfigure stuff.

The front desk staff gave me a temporary access card to get into the back room where the paging system (among other things) was. I scanned the card and, what do you know, it worked the first time they “programmed” it, which is more than can be said for other places I had been. Oh, and the doors swung open automatically, which I thought was cool, and pretty helpful if you had a rolling toolbox or a dolly full of stuff.

I went to the system, and sure enough, the little display is blank and didn’t respond to any key presses. There was a fan running, but it otherwise appeared “brain dead”. Well, replacing the motherboard would mean redoing all the configuration and such, so I turned everything off and started pulling out the power supply first. Ouch! I caught my thumb in the rack! That’s gonna leave a mark. Well, at least I’m at a hospital. So, after correctly placing my fingers this time, I removed the power supply properly without hurting myself any further. Putting the new one in, however, didn’t yield any progress. Okay, so the motherboard was the next natural suspect. I plugged it in and voila! No display, no sign of life other than the fans. How could this be? The keypad/display module had to be unplugged anyway to replace the motherboard, so if that had a loose connection, I would have already fixed it by unplugging it and plugging it back in. Well, I swapped the two power supplies between each other’s slots, in case the “new” one was also bad. No dice.

I’m out of ideas as to why the system wouldn’t boot at all, so I contacted Company B tech support. To their credit, I got a live person. Per their advice, I swapped the amplifier module: Apparently, there was digital 2-way communication between it and the motherboard, and if it were faulty in such a way as to put “garbage” on the communication line upon receiving power, it could prevent the motherboard from booting. That meant another trip out to the company car, seeing how I only had brought a power supply and motherboard.

I started toward the door I came in and was about to push the exit bar when I noticed the sign saying “DO NOT TOUCH DOOR: use sensor on wall”. I looked to the side of the door, and there was only what looked like a blank plate there where an exit button would have been. I waved my hand over it, and of course nothing happened. There was something above the door that looked like a sensor, but practically high-fiving it (without touching it) had no effect. There were also what looked like sensors on the door itself, but waving at them or doing jumping jacks did nothing.

Uh oh. I look around for other exits. There was one behind me, but it looks like I’d have to go clear around the building to get back to the car, and if the back lot was fenced in, I’d really be up a creek without a paddle. I did have my cell phone, but my eyes landed on the land-line phone in the room first. I figured it would be easier to call the front desk from that.

There was no written directory on or near the phone, and no directory on it’s little display. I tried the most obvious 0 for operator. I waited on hold for several minutes. As soon as I got someone, I asked “how do I get out of the back room without the alarm sounding?” and they were confused. I described the room I was in, and it dawned on both of us that I had reached someone in another building. I said “Can you put me through to the front desk at (hospital address) or security, maybe maintenance? I’m not trapped, but I just don’t want to trip the burglar alarm cause there’s signs telling me not to touch the door” “I’m sorry, I don’t know the right extension or who to contact for that” “Okay, never mind, and thanks for trying anyway. This isn’t your fault” and with that, I hung up.

At this point, I was just going to open the door manually and let the alarm sound. If anyone questioned me, I’d point out the process I went through to try and avoid the alarm. Just as I was about to open the door, I heard the card reader on the other side go “beep” and the door swung open. I explained my whole ordeal to the guy coming in, and he pointed out the little hand-wave sensor on the wall about 10 feet behind me, partially obscured by a shelf. Not only that, the sensor didn’t say “exit” or “do X to exit” nor was it the same color as the door or anything attached to it.

After the whole getting “trapped” in the room ordeal then being “rescued” I got back out to the car and got another amplifier module. And a Module F along with another power supply, for good measure. I planned on putting the original motherboard back in so I wouldn’t have to reconfigure everything if the amplifier turned out to be the problem. I put the original motherboard back in, then the “new” amplifier module. I left the first replacement power supply in place, just in case the original one had somehow fried other modules.

Before turning things back on, I put the antenna power meter in series with the transmitter line to the antenna. One might assume an output short circuit would be the only way to damage an amplifier, but for strong transmissions at high frequencies, an open circuit can result in “reflected” power back to the amplifier, which can also damage it as well. Therefore, it’s good practice to check for reflected power if the amplifier fails. (Better systems can detect this and shut themselves down. In fact, this system is supposed to be able to detect this, but just in case it doesn’t, it’s still good practice to check for it anyway)

I turned it on, and the original motherboard booted when I had the “new” amplifier in. So Company B tech support was right on the money. Score one for them. Good news: Reflected power wasn’t nearly high enough to cause any problems. (There is usually a small amount when the connectors are several years old, but it’s only a problem above a certain amount. Radio techs will know what I’m talking about)

So, I left the site for the day, declaring the ticket closed.

24 Comments
2024/11/01
17:35 UTC

417

It's good to be a big fish in a small pond

This one reminded me of our first experience at converting to VOIP.

When the day arrived to start replacing the 40 year old phone systems in our stores (due to replacements for broken phones costing $300 each, refurbished, at best, with a 50/50 chance they'd work), we talked to our phone (and internet) company about what was available. They were transitioning from providing analog service (over the internet, but converted on-site) to VOIP. OK, makes sense, it's the right choice.

But they were new to VOIP, and still getting a handle on how to make things work the way the customer wants, and most of their customers are offices, not brick & mortar retail. They don't know what we need, and we don't know what to ask for. OK, we'll work with them on that. They've always taken pretty good care of us. (Not the cheapest, by any means, but reliability is more important to us, because without the computers, we've got about 3 days offline before it all starts to fall apart.)

The problem was, the phones they sold us came out of the box configured for an office. The receptionist's phone rings, they answer, and transfer the call to the person it's for by extension number. Retail stores don't work that way. We need all phones to ring, be able to pick up the call anywhere, put it on hold, pick it up from another phone, lather, rinse, repeat as the associate runs around the store answering questions.

On the old phones, this is one button to put a call on hold, and one button to pick it up from any phone. The default way to do this on the new phones involves multiple buttons (including a four digit extension number) to park the call, and an equal number of buttons to pick it up. (And there's a button marked "Hold" that actually puts the call on hold - but it can only be picked up on the same phone.) Needless to say, this won't fly, even with a running start from a tall cliff. They knew this going in.

So they configured the phones to move the call to a park line with two buttons, and pick it up from any phone with two buttons. Worked . . . Okay, if not ideal. Except it relied on a feature that was deprecated, and when they did a software upgrade, it disappeared.

Then they found another way to do the same thing, but with three or four buttons. Not ideal at all, and it tended to result in dropped calls. And the particular store we converted first is in a beach city full of people rich enough to live at the beach, but not rich enough to live in the very rich beach city next door, which makes them . . . cranky. These people won't call back to finish their call, they will call back to scream obscenities at you. The only reliable way to put calls on hold such that they could be picked up on a different phone was the "official" way involving waaaay too many buttons. Plus, it also relied on a deprecated feature, and was never going to have the bugs addressed.

One of the cashiers - who has been with the company for a couple of decades and was the best they had for working with customers - simply refused to answer the phone. And her manager didn't blame her. The manager (also one of our best) was ready to quit over it. And we didn't blame him.

So a conversation was had, between me, who makes all the technical decisions, my boss, who has signature authority, our account rep (who was sympathetic, but had the technical savvy of a turnip), a senior VP from the phone company, and the lead technical guy for the phone company. It started off with my boss telling the VP "You have 30 days to make this work the way we want, or we're finding a new phone company." We were, at the time, about a quarter million dollar a year account (we're more now). Not their biggest, by any means, but big enough to get their attention. (Our account rep told me he had bigger accounts, but not many, and not by much. He was very concerned. I believe his bonuses were heavily based on total revenue generated.)

Took 'em two weeks, but they discovered some kind of macro feature ("key system emulation"?) that let them reprogram the buttons to do the complicated sequence to move the call around. One button to put a call on hold, one button to pick it up anywhere, and it worked (and we got CallerID to boot!) almost exactly like the old phones. Only difference was it was a different button to pick up the call than to put it on hold. I requested the tech guy write detailed instructions on how he did it and include it in the records for that store, so that I could refer future project engineers to that when we got to other stores. And he did, and I believe they still use those instructions to this day. And we're not their only brick & mortar customer these days, either.

23 Comments
2024/10/31
20:12 UTC

1,187

They always forget about IT.

Some years back, it was decided that our analogue phone system would be replaced.

Once this decision was made and everything signed, we in IT were notified of this change.

In that order. Yes.

My boss naturally let his many and well qualified thoughts be known, but as is common here these were dismissed. For those familiar with OFSTED, our overall rating was "Good", while their rating for Senior Management was "Needs Improvement". For those not familiar a government agency rated us as 3/4 stars overall and 2/4 stars for management (4/4 being Outstanding and 1/4 being Inadequate).

The person responsible for this was neither IT or senior management, I don't recall her role exactly now but she was the villain of many of my stories. How her proposal got accepted without our input or even knowledge would be mysterious and a cause for great concern anywhere else, but what can I say any more eloquently or succinctly that OFSTED have not?

So we meet with the supplier. Our questions are asked, and some are answered. One in particular was compatibility with ethernet daisy chaining computers with our existing setup - VLAN'd, solid and secure as it was. "Yes yes yes, all that will work". One of the techs in particular had an attitude that I could describe as "needs improvement" and customer service skills that were "inadequate". I had the strong feeling from him that he was in his very early 20s, possibly this was his first techy job, and was absolutely blindly loyal to the company having known little else in his career. His response to many of our concerns could essentially be translated to "No. Our product is good. Our product is beautiful. Our product is right, and you are wrong to question it".

I sat in on one training session. There was one member of staff in HR who I had a good relationship with and had been very kind and supportive to me over the years when I needed it, and she was always very appreciative when it was my turn to support her technical issues. We respected each other and were humble to each other's expertise, I had a soft spot for her and was always available to her - a few occasions in the fire together trying to get the monthly payroll processed with a third party on time will forge strong bonds. She was very excited and asked a very interesting, pertinent question about a certain feature. Mr Inadequate got Right. In. Her. Face. and hissed "NO! It doesn't do that!". She was absolutely crushed and I was incensed.

Do our desktops PXE boot through the phones? Do they balls. All staff are now without both their computer AND desk phone whenever we need to reimage. Mr Inadequate's response is of course to blame our network. I'm neither surprised or bothered by this, who amongst us, hey? Evasion and misdirection of blame between IT and a supplier? Bread and butter work, all the live long day. I'm not angry at Mr Inadequate for this, I'm deeply disturbed. He's not making excuses. He BELIEVES. He's of absolute faith in the infallibility of The Product. It's actually a little frightening to see the zealotry a young man can display for reselling a third rate IP telephony system.

My boss does all he can to mitigate the nightmares, there are delays and pushback from us and the general staff. Complaints roll in, we redirect everyone moaning to us in the Villain's direction and make it clear who is liaising (responsible) for all queries related to the new phone system. As we weren't consulted there is nothing we can do, there's no technical requirement to hold them to or UAT for them to complete. There's barely a week of snagging support, then we're shunted to their helpdesk for standard assistance.

The only happy ending to any of this was when the Villain who had unleashed all of this on us made a very genuine, very sincere, and very out of character apology to us.

90 Comments
2024/10/31
17:34 UTC

295

Tech-aura effect via cellphone call

A tale from before the pandemic when i was doing a lot more server maintenance.

Due to a botched update I got the fun(1) task to do midnight maintenance on one of the applications we are supporting.

Midnight hour strikes and I ssh into the machine, stop the application, untangle the bad update parts and do the needful. Pretty straight forward job, just remove a few folders and the application will recreate them correctly on startup. Time spent on this is maybe 5 minutes tops.

Now to restart the application and verify my work as good. I enter in the command to start up the application and switch to my browser to verify that it starts.

Nothing. Application URL only shows the hated nginx landing page.

Ok, something must be wrong. Let’s not panic, I’ll just restart the application. Maybe it couldn’t recreate the folders.

I restart the application again. No deal. Ok, time to do a deep dive in the logs.

I scour the logs, tailing in one tab and manually searching the same log in another tab. After scanning the log i can definitely guarantee that there is nothing out of the ordinary in them.

By this time my brain is thinking in terms most commonly transcribed in comics books by symbols most commonly found below the F-keys and the letters on a standard keyboard.

*ticktock ticktock time’s ticking*

I update the application browser page again in the vain hope it’ll work. No dice, still just the nginx landing page. After staring at the nginx page for almost 10 minutes I weigh my options:

a) Double down and bullhead my way through, hoping something i do will fix it

>bad idea, might frack it over worse

b) Leave it as is and grab SrTech first thing in the morning.

>worse idea, will generate a lot of incident reports about application being down before i get in to work tomorrow

c) Bite the bullet and call SrTech for help, waking him up in the middle of the night when i know he has a long day ahead of him tomorrow

>…oh blast. This is the best choice, isn’t it?

A few more minutes pass as i hope against hope i won’t have to call SrTech. No good, gotta do it.

I call up SrTech and the conversation goes something like this:

SrTech*: *mumbled sleepy greeting**

Me*: Hi SrTech, It’s me. Sorry for waking you up at this hour but i think i have messed up the application restart*

SrTech*: OK, walk me through what you have done*

Me*: I shut down the application, did the needful and then when i started up the application it won’t start. It just goes to the nginx landing page even if i update…*

Here i updated my browser to emphasise my words, even though i knew SrTech couldn’t see what i did.

It. Bloody. Worked.

Application is up and running as smoothly as if nothing ever happened

Me*: …um nevermind. It works now. Sorry for waking you. I’ll let you get back to sleep.*

SrTech: * mumbled sleepy goodbye *

(1) Fun in the Dwarf Fortress sense of the word. Remember: Losing is Fun.

36 Comments
2024/10/31
14:29 UTC

241

Petards that hoist people, part 2: don't dismount the scratch monkey

(Reintro: Support engineer at a company based in Seattle who is known for a tornado)

A common wisdom is to never go into maintenance without "mount(ing) a scratch monkey". There's a story to why they call it a "scratch monkey" involving a swimming primate, but the point is this - if you're going into maintenance mode, make sure you've tagged in/tagged out, signed off, opened the maintenance window, inform your users that this is gonna be a little bumpy, and you do the thing within that temporary arrangement because if you don't, you're going to blow up the pager.

Here's one such story.

A call comes in, we say hi and all, and he needs a remote right away. The colleague o' mine who owns the case is out that day. Line's noisy, so I tell him we can't get that going without a diagnostic file.

...which he...can't...get.

At this point, I started asking for a read on the errors he's seeing. It took me four tries to get it in a way he could understand - though to be fair, English is a hell of a language. But he basically started reading a bunch of daemon restarts.

...ayup, we're going to Teams.

Issue at hand is simple: after upgrading the operating system from an RMA replacement, an attempt to load the configuration backup failed for reasons unknown to me. The result is multiple daemon restarts.

We go in. I can't take control, so I watch the daemon restarts. Can't run the diag dump on the CLI, it requires a daemon that's not starting to actually be able to run. Reboot...um, well, it did work fine for all of ten seconds and then they could not get a thing started. I think now's a good time to roll back.

Talking somebody through command line is sometimes painful.

We get the CLI going, I tell him to run the diagnostic once more...and it burps. OK, let's start from the top. Let's roll back to the previous version. Run the command to change volumes and...

...hey. Hey, wait a second. Where's the other volume?

Again, three times asked - you started on this earlier version, where'd it go? Same cagey answers. And then I ask the big one.

"Did you delete that volume?"

They hesitated, and responded. Yes. Yes, they did in fact delete that volume. Somebody grabbed onto that idiot ball hard and decided it was not needed. And this is where a snippet of "Poor, Unfortunate Souls" from Disney's Little Mermaid starts playing in my head. In a fit of ignorance, they manually dismounted their scratch monkey. They blocked their fire exit. There was only one way to respond, and it required the placement of my forehead into the palm of my hand.

"I really wish you hadn't done that."

See, there are two ways out of this jam. One is to go in, review logs, and see if you can spot the bogey. This can take some time. The other is to simply bust out some bootable media and reinstall. And with this level of palpable inexperience, the decision was simple: take off and nuke the site from orbit, as it's the only way to be sure.

And I suppose it was good news for them that they could arrange bootable media and a trip to a data center.

I heard they called back, but that was the end of it from my perspective. Even so, this appears, once again, to have been a combination of ingrained ignorance combined with some unfamiliarity of the English language that tends to come up with when English is your second language - and at least one of these guys could not communicate without simplification (thus the thrice-repeated parts above) - and given that they called apparently not knowing how to boot and install despite instructions being in front, I suspect their greatest weakness was reading my language - the sort of weakness that can have you thinking Bellyvoo^1 is wee ired^23. So in my frustration, these guys have some sympathy for me - because my two native languages^4 are insane.


^1 Bellevue

^2 Weird

^3 Phonics, man, phonics. Not 100% accurate beyond second grade reading.

^4 English and bad English

41 Comments
2024/10/29
14:13 UTC

1,406

So where might I find the sun?

In my previous job, I worked as a field service engineer maintaining ophthalmic devices. In this role, I needed to be an IT specialist, mechanic, furniture mover, and truck driver. For some reason, they found it very hard to get anyone with all the requisite skills to apply! I quite liked the job at a base level, but they were screwing me over with the pay, so I left after the latest salary review wasn’t at all to my liking. But that’s another story.

One day, I was called out to repair an optical device at a hospital. The unit was a portable slit lamp—a handheld device used to examine the eye. Like most hospitals I visited, the biomedical department was tucked away in the bowels of the building, involving lots of long corridors and doorways to get through. This time, I was escorted in because I’d never have found the place otherwise.

They showed me to a bench with the offending device, and I got to work. It was a pretty simple fix—just a loose internal lens that needed to be glued in place. The issue was that the glue we used required UV light to cure, and I didn’t have a lamp with me. No problem; we usually just take the devices out into the sun for a minute, and the glue sets pretty quickly.

I looked around—no windows. I looked down the hall…no windows.

“Excuse me, I know this is going to sound silly, but where might I find the sun?”

I was directed to the loading bay, just down the hall and through a couple of doors. I gingerly carried my patient, being very careful not to bump the lens, which was positioned just right. I found the loading dock, but…no sun!

The hospital walls loomed upwards, giving me only the smallest sliver of sky. I could tell there was sunshine somewhere, but just not here. So I started walking, both hands keeping the device steady while also looking out for trucks and whatnot. Eventually, I found a welcoming beam of sunlight calling out to me. I walked into it and lifted the slit lamp into its rays like I was presenting a chalice to the gods of fire.

I stood there for a minute to ensure the job was done, trying to look casual and normal to the few people who passed me. But it’s not easy to look normal in that situation, so I just stood there like an idiot until the job was done. I found my way back, finished the repair, and tested the unit. Everything worked, and I packed up my stuff.

Later that day, I went online and found a nice, powerful UV torch that would handle the job without me roaming the halls looking for the sun like some reverse vampire.

32 Comments
2024/10/27
20:49 UTC

190

Lucky Guess or Experience? You Be the Judge!

I had a situation today which caused me a little panic, until I was able to think about it clearly.

On one of our website servers, there is a fairly strong and sometimes persnickety caching mechanism. It is so persnickety, that when we have to make an edit to a page -- such as a blog -- we have to be sure to make sure we check the page in incognito mode. Otherwise, if we are logged into the CMS and visit the page in regular mode, the update will appear, but it won't appear for others until the cache is cleared. However, I don't know what the cache retention policy is, so usually, we just clear the cache after an update and move on.

Today, a change was made to a page and it was passed over to me for my QA review, so I checked it in a new incognito session. The update had been made and everything was happy, so I reported up the chain that the update had been verified.

About 15 minutes later, the account person responsible for that website chatted me and said that she was not seeing the update. She has been bitten with cache issues before, so when she chatted me, she said that she had tried Chrome in both regular and incognito mode, and had also tried Safari. The update was not showing up on any of her browser instances.

I had someone else double-check for me, and that person was able to see the updates.

It was somewhat reminiscent of a problem I had encountered several years ago when I was at another company. In that instance, we had a weird load balancer situation, and a person would get assigned to one of the two load balancer URLs. So, instead of randomly getting Server1 or Server2, if you were assigned to Server1, it took a random, cosmic event of the universe to get you switched over to Server2. (Yeah, I know, that's not how load balancers are supposed to work. Don't care, that was about 8-10 years ago.)

Anyway, I knew that was not the issue in this case, because we don't have a load balancer, but something was preventing the user from seeing the updates, even though others could see it.

We got on a conference call and she even showed me that she was starting with a new incognito session. I even had her send me the URL she was using, thinking that maybe there were two instances of this page, but with different URLs.

Nope. Same URL, new incognito session, hard refreshing two or three times ... update still visible.

Then, she happened to mention, "I even tried it on my phone, and I'm still not seeing the update."

Everything is pointing to a stubborn cache somewhere between her and the website. She is about 175 miles away from me under a different ISP, so we definitely are not going through the same intermediate hops.

Then I asked her, "Is your phone going through your home's wifi?"

Turns out, it was, so she turned off that setting on her phone and hit the page using her phone's data connection. Hmmm ... the updates are appearing ... how nice!

From what I can tell, either her #WifiRouterModemThingie has some sort of stubborn cache mechanism, or, one of the hops she is going through has the stubborn cache.

So ... lucky guess or experience? You be the judge.

(Also, does anyone else have any suggestions on how I can check where the cache mechanism could be located? The user on the other end is not technical, so doing a tracert is not really an option.)

29 Comments
2024/10/24
18:40 UTC

956

It's too early to be having heart attacks like this..

Got an alert about half an hour ago about a server on one of our remote sites being low on storage. Pulled up WinDirStat to poke around and see if there was anything I knew I could purge, click on a large folder to see the files inside, and my remote access locks up.

"Must be a lot of shit in that folder.." I'm thinking to myself, before I get completely disconnected. Minor warning bells start going off, but Connectwise Control isn't perfect software and I've seen this before, so I wait a second, and then..

Blip, blip, blip, other endpoints start going offline. "What the hell did I even do??" I start thinking to myself. Almost panicking at this point, I swap over to our network controllers access page and pop in on the site. They're completely offline. This actually calmed me down a bit as there was no way in hell WinDirStat was taking the entire network down, and checking in on another site in the same building confirmed it was a brief outage. By the time I was done in there all the endpoints were back up already..

Too early for that sort of panic lol. How's everyone else's day been?

61 Comments
2024/10/24
15:09 UTC

439

New job role: Mathematician?

One from my education tech support days.

Two students walk up to the helpdesk, and I walk out to greet them and ask them what's going on. They told me they were having troubles doing a maths test online, so I get them to open the laptop, log in and show me what's going on.

The website they use to do the tests will grey out the boxes or display an error on screen if the internet drops out or something fails to load. It happens once in a while, so I figured that was the issue. I pull the laptop towards me and type some numbers into the two boxes. It works, and they're connected to the internet, so I ask them what the issue is because as far as I can see, everything is working fine.

They proceed to tell me that they didn't know the answer to the question, and neither did their (substitute) teacher, so they sent the students over to IT for help. They said their normal teacher didn't know the answer either when they were in class the day before, so they've come to us for the answer.

I told the kids "this isn't IT related, so I can't help you". I asked who the teacher was (they didn't know, substitute, but I worked it out later on), and send them back.

So I guess the school wanted me to add "maths wizard" to my long list of jobs that aren't my job, like "coffee machine repairman", "lockpicker", "window repairman" and "delivery boy"

44 Comments
2024/10/24
02:05 UTC

2,652

They always forget about the IT department

This one is from a few years ago but I was reminded of it today so figured I'd share.

My desk used to be near our help desk, which was handy because they could easily come around the corner and ask me questions as needed. It was also a great spot for listening for drama going on. One day I'm working and hear one of our guys talking with a client, everything was going fine until I hear him ask "Wait, aren't we in the same building? Uh, call me back later if you're still having issues."

He hung up and let us know that his caller had said her building was being evacuated because of a bomb threat, then he realized that we're all in the same building. No one had alerted us yet. We were standing there trying to figure out if we should evacuate too when I look over and see people streaming out of the fire exit just outside our office and suggested we do the same. Everything ended up fine, it was a false alarm, but one of our next projects was setting up an alert software that would notify people on their desk phones if an issue like that came up again.

111 Comments
2024/10/23
16:42 UTC

593

Impromptu tech support at the bar

This may not exactly qualify as tech support, but I'll give it a shot. I'll state I've been working in t/s since the mid-Eighties, so I used some valuable skills--mainly bullshitting* a client--in this situation.

The scene: A crowded bar--a good friend was dating the owner so I went with her to his bar to watch the Super Bowl. At the time this was one of the few bars with a big-screen TV, so the bar, while smaller and a little out-of-the-way, was a bit more crowded than usual, but not mind-numbingly overfilled like damn near any bar is on Super Bowl Sunday.

As it was, some idiot got hold of the only remote, and kept f'ing with the volume. Up, down, too loud, too soft....he just couldn't control it. The beer was flowing, and everyone was getting angrier and angrier at this guy, to the point I think someone (not me) was going to throw him out the window.

Fortunately, the time came when the beers took one of its effects and the guy put the remote down to head to the bathroom. The bar breathed a sigh of relief, when a thought came to me.

I picked up the remote, adjusted the volume to something reasonable, took out the batteries, put them in my pocket, and put the remote back down. I think the bar realized what was happening, because the chatter seemed to pick up a bit as he returned from the head.

After a few moments, he picked up the remote, pointed it at the TV, and was clearly pushing a button. It came time to bullshit him.

"PERFECT!" I yelled out after a moment or two, and he put the remote down.

Five minutes or so later, he picked it up, started pushing a button, and someone else had caught on: "PERFECT!" someone else yelled.

This went on for the rest of the game, with eventually damn near the entire bar yelling, "PERFECT!" after this guy picked up the remote. I returned the batteries to the bar owner sometime in the fourth quarter.

*By "bullshit a client" I don't mean lie to them. I mean play a little fast and loose with the truth to tell them what they want to hear, while doing what absolutely needs to be done to get the issue fixed. You know: The client says the problem is "X" and doesn't want to hear anything else, but you know it's "Y" and when they see you did something that wouldn't fix "X" you tell them it was actually "Y" but an adjusted setting at "X" will help prevent a recurrence. You know the drill: Ya bullshit 'em.

30 Comments
2024/10/23
13:56 UTC

447

One Sided Drama

I work as an IT Tech for a local council who provides us to support the schools in the area. Been working for them a few years now and while I was based at one site, I moved to current site this year. I have been to current site before to cover a coworkers holiday and worked on an ongoing issue here.

The issue was with the Drama rooms Interactive Screen. It would only ay sound out of 1 side. As stat3d, I've been here before so I have already had a go at fixing it, as well as the 2 usual techs. At one point the boards techs were called in to look and stated they would need to take the board off the wall to troubleshoot.

Yesterday I decided to have a fresh go at it. Started going through standard troubleshooting. Snugging and hugging in cables, verifying that sound output from the board co.es out one channel. Next I started delving into the board settings, only to find Audio options were almost non-existent. Until I noticed 3 dots on the channel changer. Clicking it bought up the audio options and what do I see? Balance set to Max Right.

Really?

Yes really. I changed the balqnce setting back to middle and there we go. My weird music tastes blasting out of both speakers. All the hours wasted troubleshooting this board, Internet searches, discussion s on Teams and outside techs. In the end, all it took was a fresh set of eyes and 1 Setting change.

I am both proud of myself for fixing it and deeply ashamed for not spotting it before.

Tl;Dr I can hear, I can hear, I'm going deaf.

39 Comments
2024/10/23
08:09 UTC

555

Feeling Appreciated

I work in Networking. A ticket I had earlier was involving a Network Printer. Teacher mentioned that 3 other tech people were here prior to me and they couldn't get the Printer to print wireless instead of her using the USB.

For the School District I work for. We always tried to have our Printers on the LAN Switch instead of Wireless Network just due to traffic increase across the wireless would be insane.

I get to the school. Start doing my thing. Drop was activated already. Wiring was good. I was able to get out to the internet. So I knew instantly The wireless was enabled on the Printer. Easy Peasy right.

The Teacher was looking at me while I'm configuring the Printer. Like Wow. You seem so calm while you work. Haha. I'm like well. I been doing this awhile but Got the printer switched to DHCP and all is good. After a Reboot and the right IP pulled. She was able to print wireless.

After she goes. Omg!!. You did what 3 other techs couldn't do. Thank you so much. It's the simple things you know

42 Comments
2024/10/17
20:38 UTC

297

In which the customer hoists themselves by their own petard - and a reintro

tl;dr: Wanna hoist yourself with your own petard? Trychmod -R 777 /var on for size!

So, it's been a while. About a decade ago, I was the technical "triage nurse" at $UberNetworks. Well...I'm still there. And I've been promoted - more or less "senior attending physician. And after nearly eight years in this role here, I'm quite astounded by the number of people who make me wonder how they even got into our field....

This is one such story.

We're setting the Wayback Machine to sometime earlier this year. Don't remember when. Not important. But it was an afternoon ticket, lands in my desk, and I take a look: customer having a peculiar error when he tries to ssh into the box, and he wants to know why.

My usual technique for support is to stare at a ticket, see if we have any diagnostics, get them as needed, and we did all this and then I went in.

First check: a particular file somewhere in /var/*/ssh/* was apparently set world everything. This is momentarily confusing to me, and being I'm a Linux nerd, I already know that permissions do not just change on their own, somebody has to do it...

...in the immortal words of the sage Lister, "Aw, smeg!"

OK, clearly somebody did this, this sounds like an indicator of compromise. /var/log...oh this needs words stronger than smeg. Like shit. Shit's a good word for this. Look at all those permissions errors. Audit logs...nothing here, OK, it's probably CLI and don't tell me he just....

Check diagnostics...hmm, ntp is broke, nobody's answering hails, better tell 'im.

Look at commands, and being that our product is built on a Linux box, we have a full Linux install on there. I wish we had emacs^(1), but that's another story and we have nano, I'll live. But this means we also have 'history'.

I look at the output of history. Lots there, let's do a simple text search for chmod...ohhhhhh, shit, no he didn't. Oh, my gods, he did....

There it was, like platform double suede - exactly what I was hoping he did not do, and my hopes dashed, 'cause there it was, like disco lemonade.

In the history, with a username that I could only identify as being the customer contact's username just by the spelling of it, I see what I was afraid of. chmod -R 777 /var.

I stared at my screen in disbelief for five minutes, so we're going to pause the tape here and fast forward.

See, I've been dealing with computers since I was a child whose dad bought the TI 99/4A as the family home computer. I've been working in this field since 2006 in some way or another, with the exception of two years of college. I've seen people who I can't help but wonder if they got their A+ as the secret toy surprise in a pack of Cracker Jack. And in all that time, I had never seen somebody make a mistake that is the same grade of mistake as some wannabe skr1pt k1ddi3 who was trying to impress other nerds with l33t sk1llz. Until that day. When this guy, for whatever reason, altered the file permissions for - quite literally - everything in a Linux install that could be found in /var.

The reason the file permissions were changed were because this guy did exactly that.

My response and conclusion was thusly passed via email. Not five minutes later, I get a response - a request to close, sent as I was informing his sales team.

And then I check his ticket history.

Come to find out, he opened another case for the exact same problem right after he requested closure of mine.

Double you. Tee. Eff. Is this guy even thinking? No, really, is this guy even thinking?

Oh, it's on like Donkey Kong, motherfucker, you do not get away with pulling this kinda mommy daddy game^(2) horseshit on my watch.

Ticket intercepted. Pulled in, advised closing as duplicate, do just that. At this point, the sales team has been contacted. Oh hey, they're still here. Teams time! Passed word as to the update since this point, he nods, and he's gonna call the guy after he and I talk on the phone a minute. At this point, I'm wondering to my sales guy as to what exactly would even possess somebody to do just that, like what makes someone think this is a good idea?

A couple days later, I checked back in with the salesperson. He still had his job at that time, but it took a lot of convincing ot get him to admit it and stop denying that we were on to him. As best as we can tell, he was apparently doing it to prove some kind of point about the security of the VM installation - by doing the exact things you do not do. But after the Crowdstrike incident and my hearing that nobody actually got canned from that debacle, I guess I'm not surprised that this guy still had a job at that point. But at this point, I can't help but wonder if he is considering prospects in the wonderful world of convenience stores, because that - in my book - is a potential career-limiting move.

^(1) Yes, I know, ed is the standard editor...

^(2) What's the mommy daddy game? Well...if you have kids, you've probably played this game with them, and perhaps to some level of amusement. If you don't, it's the game where a kid asks mom for something, and on refusal tries dad.

85 Comments
2024/10/17
05:14 UTC

307

We All Fall Down, and A Reintroduction

Hello again! Or maybe for the first time!

Some of you might remember me from such stories as Peanut Butter Jelly Drive, Interrupting Cow, and the Wireless Printer Before Time.

I've been quiet for a while, and I have a very good reason for that! I'm lazy!

Err, that's not a good reason... What was I supposed to say?

Oh! I've changed jobs and am now actually in Tech Support! For reals!

That's right: I'm now half of the one man, one woman crew that supports four US based plants, plus a Canadian warehouse, a Mexican warehouse, and a whole gaggle of remote salesmen, customer support, and a bunch of other characters from a few different countries! And I've been compiling stories... Most may only be posted after a good amount of time has passed to better abstract people and places to keep nosey nellies from making incriminating connections.

But that's ok, because I've been there for three years now and I have a few stories to tell.

So, with that introduction/reintroduction out of the way, have a recent story involving Big Blue, Old Data, and Money!

Time: August 2024

I've been with the company for nearly three and a half years at this point. It's been great: the people are generally great, the work is sometimes tough but rewarding, and I've really helped my boss out by taking over projects and most daily tasks, freeing her up to actually be able to take vacation and enjoy herself from time to time.

This was not one of those times.

Background: several years before my employment, the company I now work for was bought out by a multinational company because we were the leading producer of a very specific product catalogue that meshed well with what they produced.

The US operations kept the old name, and the tagline, "A xxxx Company" was added to show who our overlords were. In addition, our operations were forcibly catapulted into the modern era. New Cisco blades, switches, industrial switches, wifi APs, etc... were bought, installed, and configured. Old systems were removed or integrated into the new systems, including a huge migration of data into SAP.

Except for some custom software that did one thing: software that integrates the weighing scales into SAP. That software was written by a person who is No Longer Employed. And for technical reasons, that custom software was never adapted/rewritten into SAP, and calls for funding for a newer hardware system that could be brought into SAP had, heretofore, fallen on deaf ears.

You can probably guess where this is going...

One Thursday we get panicked messages that the scales aren't working. Weighing the raw product is the first step in production, so being able to correctly weigh the right product in the right mixture is very, very important.

I'm usually at the frontline of this, so I did the usual: since all the scales were affected I check the wireless connection between the scales and the system. MODAS clients are all up and working, but the scales can't get any data from SAP.

I quickly realize that this is going to be something I haven't handled yet, so I contact my boss. She's several hours away, camping, trying to enjoy some time off. Ha. Ha. Ha.

I tell her what is going on and what I've done. She checks a few things... The IBM server that hosts the scales program, as well as the old Power8, and some old data that hasn't been migrated, isn't responding. So she makes her way back.

Again, and I cannot put too find a point on this: not being able to weigh the raw product means that production can't happen. Once production runs out of already mixed raw product, no more product can be made. There's other work: finishing, shipping, etc..., but moving all of press to finishing will be tough.

To make the story progress faster, here are some highlights from This Comedy of Errors:

  • My boss does a 4 hour call with our contracted third party support for the IBM server. Together, they decide that the disk backplane has gone out.
  • By contract, the third party support had to have a tech dispatched with the replacement part within 4 hours of the ticket being created.
  • We work on a yearly contract that is paid once a year.
  • My boss rec'ed and submitted the bill 30 days before the bill was due.
  • Someone in billing changed the terms to 90 days.
  • The IBM server backplane went out FIVE DAYS after the contract ended.
  • By this time, IBM support was closed so we had to wait until Friday.
  • Even if we could get the third party paid, it would have been late into the following week before they would work with us.
  • We had to buy an out-of-warranty support contract with IBM, but they wouldn't be able to send a tech until the following Monday.
  • They ship the replacement part and it is rec'ed Monday morning. I note that it isn't a backplane, but the entire system board sans backplane.
  • IBM tech comes out Monday late morning. He pokes around for a while and makes a long call to the IBM Support Line and comes to the conclusion that the backplane went out (which is the conclusion we came to earlier with IBM AND the third party support), but because they sent out the system board and not the backplane he couldn't help.
  • IBM Tech found the right part and we had it overnighted to be delivered first thing Tuesday morning.
  • After talking with the IBM tech, I discovered he had to drive several hours to our location. I quickly ask if we can pay for a hotel room so he didn't have to drive home and back the next morning. "Oh, I can't come back Tuesday as my daughter is having her tonsils removed. I should be back Wednesday or Thursday."
  • My boss' heart nearly explodes. This means we would have been almost a week with no mix being made. Very Important People with Very Interesting Accents are asking questions and not liking the answers.
  • Fortunately IBM says they can send someone else on Tuesday.
  • Tuesday comes and the backplane is replaced, but the server isn't responding.
  • I tell the new IBM Tech that I'm pretty sure the previous one put the server in Manual Mode. "Huh, look at that, he did put it in Manual Mode." My boss' heart rate slightly drops.
  • It is finally up and running, but degraded because... a Hard Drive Has Died!
  • Another call to IBM is made, and they send out a replacement drive, and a Tech should be on site by Friday to replace the drive.
  • Drive arrives. Tech arrives. Tech drops replacement Hard Drive onto floor. Installs it anyway.
  • System is back to purring like a very loud kitten. For now.
  • And then most of the old scales died anyway...

Oh, at least now the funding for a new scales system has been approved!

Have a great day!

46 Comments
2024/10/17
03:27 UTC

1,138

The one where a marketing company would rather get their customer's domain blacklisted than learn to use SendGrid

I feel like I'm losing my mind.

A client of the MSP I work at recently contracted an external marketing AI Driven personalized email sales generation firm. They send bulk template emails to a list of potential customers and try to convince them to buy something. But they're not marketing, and will correct you every time you so much as insinuate they are.

Whatever. Not the issue I have with them. Because rather than send mail from their own infrastructure or a dedicated bulk sending service, they apparently require a standard licensed user mailbox to send spam generate personalized sales leads.

We warn them that this won't fly, that account is going to get blocked within 24 hours, and that the client runs the risk of having their entire domain blacklisted. Marketing company says it's fine, they've done this with hundreds of clients, including on the Fortune 500. Client says do it, boss says the inevitable stupid tax will be a good source of revenue, us techs are just paid to push buttons so we create them their account.

Twenty four hours pass. Security alert hits the queue, marketing.bozos@clientdomain.com has been restricted from sending out of 365 due to suspect outbound messages. Checking into it...the account was sending out standard boilerplate spam. We have a moment of 'I told you so,' get affected parties together, reiterate that this won't fly and recommend that they do what we told them to in the first place.

No, says the marketing company. This happens all the time. 365 just needs some time to adjust to their sending patterns. They "mimic human behavior" after all. But, we should create them a second marketing account so they can split their sends between them. This will totally fix it, promise. Argument ensues, but at the end of it the second account is created.

Twenty four hours pass. Two security alerts hit the queue. marketing.bozos@clientdomain.com and marketing.bozos2@clientdomain.com have been restricted from sending out of 365 due to suspect outbound messages. Both accounts were sending out standard spam. The 'I told you so' is said with a sigh today. We again recommend they do what they're supposed to.

No, says the marketing company. This has been happing increasingly often. What we really need is a third marketing account so they can be super absolutely sure this doesn't happen again, super duper pinkie promise. The ensuing argument has more tension this time around. A third account is created at the client's insistence.

Twenty four hours pass. Three security alerts hit the queue. marketing.bozos@clientdomain.com, marketing.bozos2@clientdomain.com, and marketing.bozos3@clientdomain.com have been restricted from sending out of 365 due to suspect outbound messages. All three accounts are sending out standard spam. The 'I told you so' is said through gritted teeth. Boss finally puts his foot down, says that we are not going to be creating an infinite series of licensed marketing user accounts. You are going to need to find both a new IT provider and a new domain at the current rate. Argument ensues, further spam sales generation sends are paused until a resolution can be reached. A meeting is scheduled.

The meeting happens, between myself, one of our senior techs/technical executive, stakeholders at the client, and the non-technical account manager from the marketing company. Account manager insists on giving us the sales pitch for their company. "We send bulk template emails to a list of potential customers and try to convince them to buy something" says the account manager in her native tongue of corporate buzzword slop. Great. Amazing. Tell us what shitty bulk sending platform you use and the spf record you want to us add and we can be done with this.

No no no, says the account manager. It's not our business process to use those. We prefer a personalized approach. You see, we mimic realistic human behavior. Our weird proprietary tool that we've grafted to this poor mailbox sends a message once exactly every 120 seconds - just like a human! We personalize our messages by using the same subject line every single time! These are not standard marketing messages, they're an AI driven, personalized sales generation platform. Transcendent. Enrapturing. You're sending spam. You're going to get the client blacklisted. I refuse to believe that we are the first people to have pointed this out to you.

Well, the account manager admits, we have been noticing these issues recently. Since last month, apparently. But we're totally 100% certain that if we just keep at it, 365 will give up eventually! We tell the client this is untenable, unsupportable, and poses a serious risk to their business operations. Marketing company refuses to budge. It is eventually 'agreed' to buy a clientdomainmarketing.com, use it to create a seperate 365 environment, and let marketing company go wild without risk of contaminating the primary domain's reputation.

Am I crazy? Does this sound like anything remotely reasonable? I feel like I'm going insane.

90 Comments
2024/10/15
13:04 UTC

1,196

That’s not my son’s laptop!

Years ago, had a college student bring by his laptop for repairs. Keyboard stopped working, according to him, and he had no idea what the cause could be.

After he left, I quickly surmised that someone spilled a sugary beverage on it, so I contacted tech support for the model (let’s say it was HP) and they quickly place an order for a replacement part. During the call, support also mentioned that a previous support call was made on this laptop for, you guessed it, a spilled soft drink. Noting that information, I proceeded with the order and, when the part arrived, swapped out the keyboard.

After verifying that the laptop was functioning properly, contacted customer to pick it up. I left it running on the repair table and moved on to other tickets. The following morning, I noticed that the screen was blank and decided to tap the keys to awaken it. Nothing happened. Listened and could hear cooling fan running, so I cycled the power. Powers back on, except the screen is still blank.

Reached out to the customer to tell them the situation and see how they wished to proceed. Here is when dad, a local attorney and expert radio/TV commentator, gets involved. He starts cussing at me and threatening me with a lawsuit if I don’t replace/repair his son’s computer. I calmly inform him that, no, I will do no such thing for a previously damaged computer. Incredulous, he accuses me of lying about previous damage to cover my ass for negligence. That’s when I inform him of the conversation I had with HP.

Now, I had him dead to rights, but this is where I was surprised. After his brain audibly glitches, he says, “wtf are you talking about HP? My son owns a Dell.” My response was that clearly there is some misunderstanding here on your part because I’m looking at an HP, not a Dell.

No apologies, nothing comes from Mr. Attorney. Instead he sends the kid to come get the laptop and pay the bill. I had to know what the hell just happened, so, when the kid shows up, I ask. He sheepishly admits that he had his frat bro’s laptop, because frat bro had broken the kid’s Dell laptop and given the kid his HP laptop. Guess frat bro never mentioned spilling a coke on the HP and this kid figured his parents would be none the wiser.

To this day, Mr. Attorney is on TV/radio to offer his opinions on whatever legal case is in the news, and I chuckle every time I see or hear him.

17 Comments
2024/10/13
22:13 UTC

471

They said it worked on Windows 95

Cast your mind back to 1995 when Windows 95 became a thing. This is set in late 95 or early 96 when new computers came exclusively with Win 95.

I had a customer who had an existing system for inventory, POS and accounting. It was a DOS based system and was written in Visual Basic. Not Studio or .net. Just visual Basic.

So customer wanted to upgrade computer and existing system to latest version. I was not a reseller for the system, but they agreed to sell a new version to me. I asked them if it worked under Win 95. They assured me 100% it worked.

I assumed they had tested it under Win 95 after such a definitive statement. I was wrong they lied.

First problem was they wanted customers details before they sold it to me. I was suspicious, so i gave them his name and my fax and phone number.

So then first problem was all the data didn't transfer correctly. I rang them and they asked for a copy of the system. They stated that Windows backup onto floppy discs would be sufficient. So backup was done and airbag to remote city.

Clue 1:

A day after they received it I received a fax addressed to customer with page after page of error messages and a suggestion that customer contact their Authorised Dealer locally. I rang them and asked politely what restore version they used. They said DOS 6.2.

We all know why they got errors don't we? Versions of backup and restore were not compatible between versions of DOS. I asked less politely why they were attempting to throw me under the bus when the mistake was theirs alone. The answer was not really satisfying. We resolved the restore problem and then they sent a new version of the transfer program which did transfer correctly.

Then the customer attempted to do the end of month on the new system. He sold things and at the end of every month he ran a summary report of everything he had ever stocked with on hand and sales per month columns. He then looked at it on the screen. On his old system it worked fine. On his new system it gave "Out of memory" errors before it finished the report. It did this even when I quit Win 95 and the underlying DOS system showed 640K free.

Clue 2:

I contacted the supplier of this program and the help person told me I had to run memmaker on the system to allow enough RAM for the report to run. Evidently every report ran in memory and had no spooling capability. I advised them that this was a brand new Win 95 system and as such did not have an autoexec.bat or config.sys. It also did not use (or need) memmaker.

The help person told me "Trust me, you just need to run memmaker" I asked them if they had run this program on Win 95 and it turned out that they had not. In fact they had a copy of Win 95, but had not installed it. They had plans to install it on one of their own computers at home. As well they had no access to the developers who existed in another country. They had no idea how to fix it or even to go about finding anything to do.

I realised I had been lied to by lying liars who lie. (pants on fire). I had anger issues in those days, especially when people lied to me. I gave them a roasting for being idiots. Unfortunately I did this in the customer's shop and I'm sure he heard me call them liars (and worse).

I told him that there was no way for me to make his system work for his report without some limit being placed on the number of items selected. In perfect hindsight perhaps some limit on vcache in system.ini may have helped, but I had no knowledge of this. In those days there was not the plethora of websites available with all the knowledge anyone could want.

I was never invited back to his shop, and I found out later that the local Authorised Dealer for that program took over that customer. I have no knowledge if they ever fixed that (unpolished) piece of .... I decided I wanted nothing more to do with them.

In hindsight perhaps I could have questioned them more about their blanket statement that it 100% ran on Win 95. Perhaps I could have been more tactful when talking to them on the phone. Part of the problem was that the customer was an hours drive from me (and an hour back) and that the main supplier of this program was 2000 Kms away from both of us. I was calling them on my mobile phone which cost me lots of money in those days and I had other customers who needed me more.

41 Comments
2024/10/13
23:49 UTC

Back To Top