I don't work in tech support, but I am knowledgeable on troubleshooting, especially when it comes to software issues. I often help friends with PC issues in a telegram group I am in.

Today, we were all discussing playing a game as a group, and someone mentioned that they can't play the game because it crashes/freezes at random. I immediately jumped at the opportunity to help, and the conversation more or less went as follows:

Me: How much RAM do you have?
Friend: I have 16GB.
Me: How much does the game use?
Friend: I allocated it 2GB. But most of the RAM is taken up by Chrome.

At this point, I'm confused. Yeah, Chrome is kinda notorious for eating up RAM, but there's no way it is using up nearly 16 GB of it. Nonetheless, I state the obvious:

Me: Then close Chrome when you play the game. Force-close it in task manager.
Friend: I don't want to do that, it takes forever to start Chrome up again.

Obviously, it won't take that long to start Chrome again, so I'm confused. I let some other friends to some tech-support-talking for a bit, and then the friend reveals the actual problem:

Friend: I have 1850 tabs open.
Me ,realizing what the real problem is: Why do you have so many tabs open?
Them: I've just done it for so long that I'm used to it.
Another Person: Dude close some of them!
Friend: I don't want to, and I don't want to bookmark them because that will take forever.

At this point I gave up and told them "you know the problem, and the solution to the problem. I can't help if you don't want to fix it" and moved on. I knew their claim that it would "take too long to restart the browser" was bogus at this point, since they were never going to close it to begin with. I will never understand how people can know the problem AND the solution to it, but still decide to ask for help, knowing full-well that they will never fix it anyway.

Hiya,
Not sure if this belongs in this sub but here goes.
As i have said before i have a real AH as a boss but to add to everything he claims to be very intelligent but I have yet to see any sign of it. AH has made many a mention in this sub

One of my tasks is to teach him how to use the new CRM that we have obtained. He also wants his own special dash board to show specific metrics. From my side no problem. I create his dashboard and get an email asking me to explain in minuscule detail what each metric means and why they differ.

There is a graph that was created to show various leads and who captured them and when. Not a difficult thing to understand. AH send me an email asking how many leads were created in a certain week. Not difficult look at the graph, place you mouse over the key and it will show a dynamic result. I prepared a response explaining how to put the mouse over the different parts of the mouse, what each response will be.

Then I got another email asking question, that if i didn't know better would have thought came from a kindergartner. I then took the first email response and asked an AI how would i answer the question if it was a little child. Here is the response

​

Imagine we have a colorful calendar that shows how many leads we get each week. Each week has a different color, just like your crayons! Week 49 is special because it's red, like your favorite fire truck.
If you look at the calendar, you might see two red blocks. Those blocks show how many leads we got during that week. It's like counting how many toys you got for your birthday!
To see how many leads we got from each part of the school, you can hover your mouse over the colored blocks, just like pretending to fly over the calendar. This will highlight the blocks for that week, making it easier to see.
And if you want to know more about a specific part of the school, you can click on it. This will open a special box, like a treasure chest, that tells you exactly how many leads we got from that part each week. It's like counting all the different toys you got from your friends and family!
So, the colorful calendar helps us see how many leads we get each week and where they come from. It's like having a special tool to understand how our school is growing, just like you grow taller each year!

​

My dilemma.... send this or not?

I am posting on behalf of my friend, A.

A isn't an IT worker but they work in an accounting firm, and this awfulness happened to the poor IT worker there.

A's Supervisor (K) has a company laptop. This laptop decided to die in the middle of the work day. Just out of nowhere- ZOOOOP, black screen. Charger LED was also not turning on anymore, so K decided he needed to bring it to IT.

However, because his laptop was so dirty, he felt he should clean it up a bit because otherwise, IT would blame him for the laptop breaking during use. It wasn't as bad as some of the nightmares we've all seen, but it was covered with dust, streaks, pet hair, etc. And obviously it was definitely his fault the laptop wasn't working anymore, as he clearly didn't take care of it.

A watched as their Sup cleaned his laptop using paper tissues and his own spit. And then he hands it to IT.

So today started off normal, until my boss called me and says the police have been trying to reach us about some 911 hangs that came in and when they called the number back it went to a fax machine.

first red flag, our emergency notification system didn't have record of any 911 calls placed that day. Second red flag, 911 and fax should never be in the same sentence, thirdly, i don't even recognize the number they are referencing.

So that's weird, i tried calling the number on the routers pots line and it dialed through just fine, got voicemail, but not a fax machine. Had a local user place a test call and confirm the call came from us, so the number is ours.. (guess i should add more 911 info to our system documentation, i took over when previous guy left, no telephony experience, there are still a lot of new to me things that creep up) Ended up calling 911 service center, supervisor confirmed the test calls we made matched the earlier hang ups, then he played a recording where we could i got more info.

I think there was a miscommunication, the cop didn't say he got a fax machine when he called the number, he tried to say it sounded like it was a fax machine that placed the 911 call. The machine called once, couldn't handshake, then tried again after a couple minutes. So..2 calls to 911 20 seconds long waiting for a handshake.. oof.

so now i'm back to trying to figure out out someone in our office managed to hook a fax machine up to our router, it would require going into the closet and unplugging the line from the router into a fax machine.. so.. yea, that didn't happen. I considered aliens for a second, but also, not likely.

it was at this time i finally came around to pulling up the timestamp of the original call in real time monitoring tool (cisco), where i finally saw the extension that placed the offending calls... an extension belonging to a fax endpoint.. at a different office, in an area serviced by a different 911 service center.

looked up the endpoint settings, the calling search space was set to the first building, hence the routing through that office.

After talking to users, this fax used to be in building a, but was moved to building B, search space never updated.

The end user to placed the fax did so using network fax printer queue, so they were trying to long distance fax, prefix with 2 1's, and didn't verify before clicking okay, they had no idea they told the machine to call 911.

Updated the call space, so at least if user errors again, the call will ID to the right building. Secondly though, had to address why no one was picking up the cop's, or my calls to site a, looked around, the call is routing to an active end point, but the office staff don't here anything routing, updated call manager to route to someone who will actually answer their phone..

While this situation can't happen again, i still don't know why my notification system didn't report the 911 calls made from the fax machine but reported the other test calls just fine, this is the only time i can remember a call not reporting since we implemented the notification system.. but that's tomorrow's problem.

So alls well that ends well, cop and 911 supervisor were understanding and pleasant to speak to, so that's something.

This is a follow-up to Yet Another IBM Upgrade, which was told in two parts.

Part 1

Part 2

In parts 1 and 2, I told the story about the problems we had installing Unix on an AIX platform. Once we got the OS installed, I was able to port our software to it without incident. This is what happened when I deployed our software on the target machine.

We were performing on an RFP for a huge government contract for an institution that sounds like "Pie R Us" if you hold it out at arm's length and squint with one eye. The plan was for me to fly out to a suburb of Washington, DC, install the software during "Integration Week", stick around to provide any integration support and make tweaks to the software if necessary, and then fly back. Easy peasy.

I flew out from our offices in Los Angeles on a Sunday and was at the facility bright and early on Monday morning. They already had an AIX workstation set up for me, and I had already confirmed with the Integration Manager and the main sysadmin that they had a tape drive -- and the necessary space on all the drives -- for me to install our software and compile it in the environment.

I was pleasantly surprised to find that the system was, indeed, installed, had a tape drive, the tape drive worked, and they had all the disk space I could ever need for the software. In fact, things were going so smoothly that our program was one of the first of several major software packages that were to be integrated as part of the proposal to the government client, and I was done with all my work by noon on Tuesday.

I spent Tuesday afternoon helping some of the other teams in testing their software, answering questions about our software, and actively monitoring the network using our network management software. There is something absolutely satisfying (to a geek like me, that is) to watch people unplugging and plugging cables into the back of a network device, seeing the port indicator go off "off" to green (or yellow, if there were significant packet loss issues) on the device, and seeing the same simulated lights on our management software displays go from dark gray to green or yellow. In other words, our software was working as it should, and actively monitoring the network as it was supposed to.

I called my boss (the primary founder of the company) and told him the good news. He then told me that due to intellectual property concerns and copyright issues, under no circumstances was I supposed to leave the source code on the network. He stressed that I had to leave the binaries and configuration files, but I had to remove the source code hierarchy.

OK, boss, whatever you say.

I told the Integration Manager that I was going to remove the source code, and he understood. I put in a new tape and made a backup of the entire source code hierarchy. After making the backup, I then restored it on another filesystem and ran a couple of utilities to compare the two build hierarchies, just to make sure the backup worked.

After confirming the integrity of the backup, I then made a second backup of the source hierarchy and did another comparison, just to be safe. By early afternoon, I had put one of the copies in an overnight delivery package and sent it back to the office in LA, and I would keep the other tape with me in case I needed anything the rest of the week. Of course, I had my original source code tape, but having the backup of the hierarchy after doing all the compiling and building was far better if I needed to restore the environment.

After checking with the Integration Manager and the main sysadmin that it was safe, I then removed the source hierarchy.

I spent the following day and a half helping out where I could, and by Friday, the entire system was nearly ready, with only a few peripheral components still needing to be integrated and configured. There was a final meeting with the Integration Manager, the sysadmin and his sysadminions, and all the vendors to go over the final things before the presentation for Pie R Us on the following Thursday or Friday.

When it came time for me to give my final status report, I iterated, reiterated, and then (for good measure), stated once more unequivocally that our source code had been removed from the filesystem and the only evidence that I had even been in the building was the binaries for our software and the associated configuration files. I stressed that due to legal issues, we could not nor would not be providing our source code in any shape or form, and that they were responsible for backing up the filesystem that held the binaries.

I was assured, reassured, and then told rather impatiently something along the lines of, "Yeah, we know, we'll be backing up the system as soon as the final integrations are made!"

I offered to make a backup of the binary hierarchy, but, since other vendors had their binaries in the directory, I was not able to make the backup because of the other vendors' intellectual property. But, I was told again that they would make backups so everything would be safe.

After a long, intense week integrating with 8-10 different vendors, everyone was tired and cranky, but were feeling pretty good about the prospects of winning the huge government contract. If I recall correctly, the other consortium bidding on the contract was actually being led by, you guessed it, IBM.

I flew home that Friday and had a wonderful, well-deserved, relaxing weekend with my family. I was pretty tired from the cross-country flight and the jet-lag was messing with my body.

On the following Monday, I got into the office and opened my email. And nearly wept.

Someone had made the decision that before building the final system that was to be used for the proposal demo, they should make sure the machine was in a pristine state, and so it had been wiped. Somehow, the sysadminion that was told to make the backup must have heard, "pack up" or "crack up" or "your momma's messed her back up", because -- you're probably way ahead of me here -- they had not made a backup of the system before wiping it, and could we pretty please send a tape with our software so they could remake it?

The company founder said, "Absolutely not!"

The only option was to fly out there and remake the software on their system again.

And so, after calling my wife to repack my suitcase and bring it to me, that night I was on a red-eye flight from Los Angeles back to Washington, DC.

(Quick side trip: I can't sleep on airplanes and was stuck in a window seat in the last row of a section. Because of the emergency exit directly behind me, my seat didn't recline. To make it even more fun, the guy in the aisle seat got a drink shortly after takeoff, took two sips, and then promptly fell asleep with his half-finished gin-and-tonic on his tray and his hand holding the glass steady. For the whole flight. The flight was full and I was unable to get up and move. So, for the entire flight, I was unable to sleep, unable to recline my seat, unable to get up to use the toilet, and had to keep an eye on his drink to make sure it didn't spill.)

I took a shuttle from the airport to the hotel, arriving at the hotel around 7:00 AM. I got checked in, and took a long, hot soak in the tub just to try to be ready for the day.

Around 8:00 AM, the Integration Manager met me for breakfast and then took me back to the integration site. I loaded the tape with our source code and ran the make, which finished in about 10 minutes without any error messages because I knew how to check /tmp first.

After copying the binaries and configuration files over to the integration machine and testing it, I looked at the Integration Manager and said, "If you don't mind, can you please make a backup or two before you take me back to the airport?"

After the backups were made and confirmed, I once again removed our source code, and we left for the airport. We stopped for lunch, and I was back on a plane by mid-afternoon. That flight was also full, and the temperature in the cabin was hotter than usual.

I think the combination of stress, flying coast-to-coast twice in less than 24 hours, the heat and sweat, and eating several gobble-gulp-and-go meals played havoc with my body, because the next day, I had broken out in a rash from head-to-toe.

Thus endeth the story of the AIX and pains associated with that particular chapter in my life.

"But, BobArrgh," I hear you cry, "what about the contract with Pie R Us?"

Why, it was won by IBM, because, of course it was!

Hello reddit, i am back with a new reddit post.Setting: So we just swapped from Google Workspace to Microsoft 365 for the company. In that phase we closed down access to google workspace and made the accounts use Outlook (E-Mail/Calander/Contacts etc.)

So we do have a self developed planning tool which we sell to customers. This program uses a service account to send mails. All worked fine, cause we used a config server which had a SMTP user with password... well safety wrong.... So a coworker asked me to look into the matter to do a direct send over the SMTP connection. Convo:

CW: Hey can you look into ticket XXXX, we want to change the auth method to direct send, it will only take you 30 min. I already tested it and it should work fine.

ME: Sure, if it only takes 30 min, I can squeeze it in

So I talked to the creator of the ticket. He does not want the password to be used -> so we use direct send. Seems simple.

After setting up a test environment, I began to test. Mails to our own Mail System, To Web.de, To Google, etc.

Well, our system worked fine, but everything else did not work. Everywhere else, it got marked as spam. Header looked fine, E-Mail Body was not at fault ... Enter me searching for hours to find the mistake.

Well, for the people who don't know. Microsoft 365 Business uses a connector in the exchange admin center, which you need to config. It was configured but the IP address was wrong... How can MS send mails with the service account, without using any authentication? (If somebody knows, please let me know)

After correcting the error, the spam mark disappeared. Still wondering how the CW tested it and got it work... but well it works now.

​

EDIT: Erroneus just made me aware that its not called direct send. It is actually an SMTP Relay with an connector. Do to the connector the auth of the sender can be varified. Thank you Erroneus

See also: https://support.insight.ly/en-us/Knowledge/article/2722/Configure\_a\_connector\_to\_send\_mail\_using\_Microsoft\_365\_or\_Office\_365\_SMTP\_relay/

​

(Sorry for the delay, but according to the clock, I have 59 8 minutes left before yesterday's "tomorrow" becomes "the day after tomorrow".)

Part 1: Yet Another IBM Upgrade (Part 1).

The next day, Steve shows up at the office bright and early with the CD-ROM drive. He plugs it in, boots the hardware, mounts the drive, and starts feeding in the CD-ROMs. He's working at a side table in my office where I kept the machines I was porting to, and I am working away at my own computer (a Sun workstation, the flavor of which escapes me -- it may have been a SPARC. We affectionately called it a "pizza box" because of its size.)

Our company was developing software to control SynOptics devices remotely using SNMP (Simple Network Management Protocol). It was actually pretty cool; our management software displayed the SynOptics hardware and showed all the blinking lights on the various devices. It was all really state-of-the-art, built on X/Motif on Unix. I had various SynOptics devices in my office and could unplug cables from the box and have the lights on the software display blink out and pop up alerts. (Ho hum, I hear you yawn; routine stuff these days, I know, but in 1989 or thereabout, it was pretty dang sexy.)

I was clacking away on the keyboard, and had multiple terminal windows open: vi in one, various makes running on the others, all kinds of things happening. Steve asked me what I was working on and I mentioned that I was porting our software to a Compaq running Unix System V. I asked him how it was going, and he said, "Oh, it's installing just fine."

I got up to get something from the printer, and when I passed by his workarea, I noticed a bunch of error messages on the screen. I don't remember what the error messages said, but the most noticeable phrase I saw was something about being out of disk space. I stopped and took a closer look, and, indeed, every command being executed was failing because there was no room on the disk.

I said, "Umm, Steve, it looks like the installation is having a problem," and I pointed at the screen.

Steve looked at me with an extremely condescending look on his face and said, "Oh, that's not an issue. That's a 'make' file, and it has all the commands necessary for installing the operating system."

I said, "Yes, I know what a 'make' file is, but I also think the system is telling you that it is out of room on the hard drive. Looks like /tmp is full."

He sighed, and then looked at me and got a very patient look on his face. At this point, I should point out that Steve was probably in his late 40s or early 50s, and I was in my mid-to-late 20s. It was very obvious that he assumed that because he had been an IBM field engineer for 15 or so years, he knew absolutely everything that needed to be known about computers. After all, IBM invented the things, right? (Well, not really, but let's just say they did.) Besides, he was clean-shaven and wearing a suit, and I was just a young, bearded geek in jeans and a T-shirt.

"BobArrgh, it really is OK. These 'make' files are really incredibly complex and were put together by our system administrators. They know what they are doing, and, to be honest, you really shouldn't worry. It should only take another 30 minutes or so and I'll have you up and running."

"OK, then; I guess I'll just continue working on porting our software to this Compaq."

I then went into my wife's cousin's office (he was one of the founders of our company) and told him, "This isn't going to work. I keep telling Steve that he probably hasn't mounted a filesystem correctly, because the system is out of space and, of course, none of the compile jobs are working."

My wife's cousin came into office and peeked over Steve's shoulder.

"Hey, Steve ... did you mount the filesystem on the mount point correctly? Because it looks like you don't have enough room for the compiles to finish."

Steve looked at him and said -- with more than a hint of impatience -- "It's fine, you see, these complex 'make' files can seem rather intimidating, but trust me, it's working just fine."

My cousin just said, "Trust me, it isn't!" and walked out.

After two hours of running makes and not being able to get any part of the operating system working, Steve threw in the towel and said, "I'm not sure what's happening, but I'm going to have to call my boss and have him troubleshoot."

About an hour later, two suits show up at the office: one was our IBM Sales Rep and the other was Steve's boss. Steve caught his boss up on the situation, his boss sat down at the computer, took one look at the screen, and said, "Well, here's your problem, Steve ... looks like the filesystem wasn't mounted correctly and there's no room on /tmp for the compiles to finish."

I peered over his shoulder and said, "Huh ... very interesting. Out of space on /tmp, is it? That's incredible you could find it so fast!"

Steve mounted /tmp correctly (stop snickering, get your mind out of the gutter) and restarted the make process. It was very clear from the way the build was proceeding that things were progressing just fine and there were no more filesystem issues.

I was still at my computer, Steve was watching the incredibly complex 'make' file spit out its normal messages, and his boss and the Sales Rep were talking quietly, looking over his shoulder.

(Slight digression. Some months prior, I had heard the following joke: How many IBM engineers does it take to change a lightbulb? Three: one to do the work, and two engineers to swap stories about lightbulbs they had changed for other clients.)

But that's just a joke, right? There can't be any truth to jokes, can there?

Well, right about that time, I heard the Sales Rep and Steve's boss talking about problems they had encountered at various times in their career, from System/36s and System/38s, and with the relatively new AS/400.

I guess that most jokes actually do have a kernel of truth in them.

As Steve found out while trying to build the kernel of an IBM AIX machine.

(Thanks for letting me stroll down memory lane. Coming soon, what happened when we finally got our software ported to the AIX.)

(And I got this posted before time ran out on yesterday's "tomorrow".)

I was sitting in my office one slow day and the CEO walked in (always a pucker moment even though I'm on VERY good terms with him) and he handed me an (obviously non business) laptop and asked if I could get it back up and running as it had "very important thing" on it that was needed shortly and as I wasn't doing anything and he still signed my paycheck, I said "you got it sir!"

Later a middle mangler comes in and asks me to do something and I say "you're number 2 in line" and pointed to the (obviously not corporate) laptop I was fixing.

Mangler didn't like that much and demanded I call whoever and tell them that it wasn't acceptable for me to be working on personal equipment.

So I pulled out my cell and called.

Mangler could only hear my half of the convo but what he heard was something like

"hey Tom, I got Bob here telling me it isn't acceptable for me to be working on your personal stuff and he wanted me to call you and tell you...

"Oh sure, he's right here, hang on"

and I handed the phone to the mangler

mangler started off saying "It's not accepabl..." and then his eyes got REAL BIG and the rest of the conversation was "yes sir" and "no sir"

He hung up and thundered "why didn't you tell me it was CEO's laptop?!?!?"

and I said "you didn't give me a chance and demanded I call them RIGHT NOW!!!!"

I got done with CEO's project and went to middle manglers office to see what he needed and his reply was "my cable box isn't working on my tv right now"

He couldn't understand why I cracked up laughing.

"lemme get this straight, you were complaining because I wasn't working on corporate equipment and you needed to be first in line because YOU COULDN'T WATCH TV?!?!?!"

I switched inputs on his tv and walked away.

Worked IT at company previously that had managers that took upon themselves to hire interns without talking to HR first and then swing by IT with the intern on their supposed first day and demand useraccount and email setup and a laptop provided at once because the intern had to start working on an important task imediatly

For the sake of the story you need to know that the CEO also was the founder and owner of the company.

One morning the CEO's assistant swings by IT with panic in his eyes and the CEO's laptop in his hand. Turns out Windows decided to mess up the boot record and wouldn't boot, a not uncomon problem with Windows XP in those days. Problem was that we had a third party hardrive encryption and you had to decrypt the drive before you could CHKDSK on the drive and fix the boot issue, not hard but timeconsuming on 5400rpm physical harddrive.

The biggest issue was that the CEO was going to hold a presentation at a university seminar in 2 hours and the Powerpoint presentation was on the harddrive in his laptop. I told the assistant that in worst case I can access the drive with the emergency disk and copy the presentation to a USB-stick and then they could run the presentation on the assistants laptop instead, because decrypting the disk and getting Windows to back to working order in less than 2 hours was cutting it close.

I copy out the presentation to a USB-stick and gives it to the assistant and he goes to set his laptop up to be used for the presentation.

Enter some young guy I never seen before.

$intern: Hi I'm the new intern and I was told you have a laptop for me.

$Me: Sorry but I have got no information at all about anyone new coming and I really can't do anything about it right now as I'm by my self today and I'm working on an urgent priority one case. I can have a machine up and running during the afternoon at the earliest.

$intern walks away and I continue working on getting the CEO laptop back in working order. Cue 20 minutes later, the $idiot-manager enters IT with $intern in tow.

$idiot-manager: This is $intern, must have a laptop and email now!

$me: Did you send a request to helpdesk or HR about this two days before hand as per policy?

$idiot-manager: No...

$me: Well then it you'll have wait until afternoon as I have more urgent matters.

$idiot-manager: That is unacceptable! $intern need his accounts an laptop, I'e brought him in to do urgent work for me...

By now I had less than an hour before I had to have the CEO's laptop in working order and sent off in a cab to the University. And I was very fed up with this entitled idiot of a manager.

$me in a snarky tone: See this laptop? This is the CEO's laptop, in less than an hour this laptop has to be in working order and at the University for him to hold a presentation. But I tell you what, if you call the CEO now on speaker phone and explains to him that your unannounced intern getting his accounts and laptop setup take priority over the CEO getting his laptop fixed before his presentation at the University and he agrees to that. Then I will gladly switch to setting up your intern instead.

$idiot-manager:...

$me: If there were nothing else, please excuse me but I have to get this laptop working.

$idiot-manager leaves IT with $intern muttering something under her breath.

40 minutes later and I have the CEO's laptop back in working order just in time for the CEO's assistant to collect it and jump in a cab to the University.

Later got a thank you for all your help from the CEO. And $intern was setup and could collect his laptop when he got back from lunch.

As they say "Bad planning on your part does not constitute an emergency on my part."

A recent story by /u/Skippy8898 reminded me of an IBM "upgrade" (well, "installation", actually) I experienced. Rather than putting a long comment in that thread, I decided to post this as a separate story.

(I wanted to call this post "YAIU" in the spirit of "yacc", but thought that might be a little too esoteric. But, if you know, you know!)

Back around 1989 or 1990, I worked for a startup company that wrote networking software that ran on various flavors of Unix. I was one of the four developers in the company and was also responsible for porting our software to any hardware platform that ran certain flavors of Unix. To put it mildly, we ate, drank, slept, and breathed (through our shaggy, bearded mouths, obviously) Unix. Our company founders knew Dennis Ritchie and Ken Thompson. One of our founder's brother-in-law invented a hugely popular programming language that powered the internet in its early days. So, yeah, we were fairly knowledgeable experts in that operating system and in creating software, make files, tar libs, etc.

Then, IBM released its AIX platform, and, somehow, we received one of the earliest models. Four boxes arrived at our office and we were told not to even take the components out of the boxes, because a certified, IBM system engineer would be there the next day to set it up, install it, and make sure we were properly trained. OK, whatever.

Since I couldn't take anything out of the boxes, I did take a look at the packing manifest and was counting the boxes and mentally checking things off. The manifest just had a listing of things we were getting.

• Big box (#1 of 4), probably the main system -- check.
• Medium box (#3 of 4), rather heavy. Hmmm, could be either the tape drive or possibly the CD-ROM drive. (One thing to note is that CD-ROM drives were not really ubiquitous at that time; in fact, I think this was our first system that used CD-ROM technology. We mostly used small tape drives for data storage.)
• Small box (#4 of 4) about the size of a shoebox but only a couple of inches tall and fairly light ... hmm ... (looks at manifest) ... ah ... CD-ROMs including the Operating System and Documentation Library -- check.

But, where was box #2 of 4?

I called our IBM system engineer and said, "Steve, just to let you know, the manifest says there are 4 boxes, but only 3 of them came." He assured me it was probably just an oversight and not to worry because this brand new system from IBM had everything built in and didn't need any anything else.

He shows up the next day and starts unpacking the boxes and setting things up.

First, the big box. Yep, that was the main system. Then the medium box, which turned out to be an external tape drive. Then the small box, which was, indeed, just a bunch of CD-ROMs. I'm thinking, "Well, cool ... the CD-ROM drive must be built into the system. That's really neat! Way to go, IBM!"

Steve starts hooking up the system and then pauses and looks at me: "Umm, where's the CD-ROM drive?"

I shrugged and told him, "Maybe in box #2 of 4?"

He said, "Oh, well, not to worry, we can install the OS using the tape drive."

I looked around, but did not see a tape cassette in any of the packing materials. At this point, Steve got a weird look on his face.

"Ah, well, I think we might have a problem here. It seems we have a bunch of CD-ROMs but no drive, and a tape drive but no tape. I'm going to have to come back tomorrow."

I shrugged, "Oh, well, things happen. Do you happen to have the documentation so I can start reading it?"

He pointed at the pile of CD-ROM disks and says, quite proudly, "Oh, the entire Documentation Library is on those CD-ROMs!"

I looked at him and asked, "And exactly how am I supposed to access the library without a drive? Don't you have a printed copy?"

Steve just said, "Well, once you have the CD-ROM drive, you'll be able to read the manuals. Anyway, I'll see you tomorrow."

(Part 2 coming shortly.)

​

Thought of this from a few years ago.

Me: Me MG: Marketing Guy

Me: I see you opened a ticket about the emails you send being marked as spam.

MG: Yes! I keep hearing from people that they've found my email in their junk folder! Fix it!

Me: Let me take a look.

I pop on the google workplace mail log reporting tool and check his outgoing emails.

They are being send out properly, passing DMARC, etc.

I pull the full original email from the system to check it out.

Everything looks okay but as I go through the body of the email I notice there's a tracking image embedded in it. The URL of the tracking image is blacklisted everywhere.

We think MG has malware or something so we start digging. Nothing comes up on scans.

We disable all extensions in Chrome and.... The problem goes away!

After checking each extension, we realized the MG, because he "needs" to know when his emails are opened, has added an extension that adds tracking pixels to outgoing emails and gives him reports.

The Extension was using several different domains for the tracking and one of them was the one on everybody's blacklist.

Disabled the extension permanently, MG not happy but not his emails aren't getting rejected anymore.

Work bell tolled at 5pm on the turkey day weekend. I was free. Or so I thought.

9pm rolls around, and I get a call from one of our good techs. So the client i'm attached to, has lots of contracts with lots of suppliers. This time, it was a billing and management vendor.

Sebastian: Hey, Adella from the Atlantis office called because the SQL connection to the Triton Database dropped.

Nerobro: Huh. We.. don't support Triton, we only have a tunnel open to them. I wonder what's up. *noises of Nero getting computer out*

Sebastian: Oh no, I'm sorry, I shouldn't have called if we can't do anything.

Nerobro: No... you did the right thing, now it's not your responsibility, and the decision is ~mine~. You did it right.

So I dig into it as far as I can. By the time my computer is up, and i'm in the ticket Adella already e-mailed saying the connection came back up. Other than giggling at the SQL connection names, and like, things that seemed like misspellings of the SQL connection. TritonWorld that was spelled TrytonWurld... Since it came back up, I decided not to chase that thread.

Atlantis doesn't do the turkey day thing, but Triton, hosted in the US, does. The outage was after work hours, and came right back up. I explained to the customer it was likely the vendor doing updates on an evening they expected nobody to be working.

That was exactly the effort they were getting for after hours work, for a system I don't have access to, and was already back up.

I am working today. So I called the vendor... and after some phone tag, it turns out, I was right. Though, since I wasn't the actual customer, it was a really weird call. Also.. I never heard back from Adella, ever. I wonder if water shorted out their pc?

Honestly my title can be the entire post and everyone in the know will shudder and cringe at that one with zero elaboration.

Not too long ago we switch 2FA to MS Auth as the other one was less secure and we kept having annoyance intrusions.

Annoyance intrusions is what my job calls it. Person A has their account creds compromised and 3rd party actor tries logging in. They are hit with 2FA and decide to try their luck. The person who owns the account thinks nothing of it and ignores the prompt or hits no.

The 3rd party actor tries again and again until the person finally gets annoyed and hits yes to shut their phone up.

After years of dealing with these kinds of intrusions, we convinced the higher ups to switch to MS Auth.

Actually thats a lie. MS Auth is cheaper and thats how we got them to approve the switch.

Anywho. We made the swap last year and we kept running into something I call Helicopter Manager Syndrome. The manager would setup his/her entire staff on the 2FA for them. They would not have their workers grab the MS Auth app from the play/app store. They would just set it up for them and use secondary authentication methods. IE Text/Call methods.

Welp Fast forward to this year and new security policies are in place. Malicious 3rd party actors are able to intercept calls and text messages logging into accounts and compromising our network.

Now it is app only. If you forget your phone? Guess you gotta drive home. Your phone is lost/stolen/destroyed in a horrible paddle boat accident? Gotta get a new phone.

Now I tell you that story to tell you this story.

Let me introduce you HMS (Helicopter Manager Syndrome) Karen. Karen is a manager of over 150 underlings whom she treats like her children. Her perfect little angels need her to do everything for them.

See since the plague wiped out most of humanity and we all started to live in underground bunkers, or just permanently worked from home, HMS Karen was always a bit extra when it came to her hovering.

If one of her underlings called into the help desk, she had to be 3-wayed onto the call.

Her staff needs warranty work? Better write up a 4000 word essay to explain why or she wont approve it. Actually that one was easy as managers dont approve warranty work and can not interfere with that.

HMS Karen was the manager no one wanted to work under, yet was the only choice due to location.

So the day comes which we send out the warning email stating that text and call methods will NOT work for logging into our systems any longer.

Then the second warning. Then the third... Yup all ignored.

So finally the day of the switch over comes and HMS Karen is calling into us frantic. By this point, Karen has lost over 60 percent of her underlings due to the economy.

$HMS Karen - You have to undo the change. We can not use this horrible app.
$Me - Thank you for calling IT this is Lightning. How may I assist you?

Small silence.

$Me - Hello?
$HMS Karen - Can you hear me?
$Me - Yes I can hear you now. Thank you for calling into IT this is Lightning, how may I assist you today?
$HMS Karen - I just... Nevermind. You have to undo this horrible change. We need to be able to text to log into our accounts. This app is horrible.
$me - I understand it can be a bit of a pain to setup, but once its up and running it is good to go.
$HMS Karen - NO its not. Its popping up with full screen ads and not letting us authenticate to log in.
$me - Uhh...
$HMS Karen - And it cost 40 dollars. Do you now how expensive it is for me to pay 40 bucks for 47 employees?
$Me - Well I have some good news there. It is actually free. If the app you have is saying it costs 40 dollars, it is not the correct app. Also MS Auth does not have any ads. So that is not the correct app. You dont have to pay for it.
$HMS Karen - That isn't true. I am looking at it right now on the play store. Its called the authenticator app. It has a lock with a keyhole in it.
$Me - Mam MS Auth is free. It doe not cost 40 dollars. The one you are looking at is a fake provided by a malicious 3rd party trying to steal your login creds.

Long pause.

$me - Have any of your guys tried to login to the app?
$HMS Karen - They tried but it wouldnt work with the QR code prompts from the logins.
$Me - So you are telling me that all of your employees have entered their UN and PW into this app?
$HMS Karen - They tried to, but it doesnt let them login.
$Me - But they physically entered the infor
$HMS Kraken interrupting me - I JUST SAID IT WOULDNT LET THEM LOG IN!!! WE DO NOT NEED TO ESCALATE THIS!!!!

While having this conversation, I am on our chat programs with the security department.

$Me - Hey... I am on with office 666, you know HMS Karen's office?

$Sec - ... Dont ruin my day please.

$me - You know those fake apps that are charging 40 dollars and stealing accounts?

$Sec - ...Thank you for reaching out to the security department. No one is available to take your call at this time.

$me - Bro...

$Sec - ok. Yeah we know the app. Its been all day with this crap.

$Me - So you know how HMS Karen is the most helicopterist helicopter to ever copter her underlings?

$Sec - English please?

$Me - Ill order us some wingstop. But yeah her entire office bought this 40 dollar app and entered their creds into the app.

$Sec - ...Didnt I just tell you not to ruin my day?

$Me - Shut up. Im paying for wingstop.

$Sec - OK. Ill get on the horn with Karen's boss and the CIO. Let them know that jimmys about to be rustled.

Right around this time.

$HMS Kraken - DID YOU JUST DISABLE MY ACCOUNT!!!
$Me talking really fast - Per security policies, I have informed the security department of the possible intrusion. Everyone in your branch has had their accounts disabled for their protection. If anyone of your employees use their our company PW for any of their non work accounts, it is suggested to immediately change it.

In my chat with security the CIO was invited in as well as Karen's boss.

$CIO - Hey invite me into this call.

$Ultra Karen - Yes me too please.

SO I invite security, the CIO, and Karen's boss into the call and "accidentally" disconnect myself form it.

$me - Oh guess I accidentally transferred instead of added. CIO you have the call now.

$CIO - OK. That works for me. Mistakes happen. Not like you could have done anymore anyways.

In a private message from CIO.

$CIO - Smooth.

HMS Karen's entire office was down that day and it took the security department 4 hours to setup their office on the correct MS Auth app. Cherry on top. CIO ended up footing the bill for the buffalo wings. Although he ordered from BWW instead of WingStop. Not my cup of tea but I wont complain about a free lunch.

TL;DR at bottom.

I worked help desk for the IT department at my university. Paid well enough for a student, and I got sharply discounted tuition, plus my folks let me just stay with them for a pittance of rent, since we lived like five minutes away from campus. I could walk to class. It was the perfect setup!

Anyways, I'm on-shift one afternoon during the first week of school one semester, students were still getting their dorm rooms setup and connecting their gaming consoles or streaming devices, etc., and so this week was typically busier for us at the help desk. We typically had to do a lot of allowing MAC addresses to bypass our NAT restriction using a web form a student filled out, or as a walk-in to the help desk. So, you can imagine we had a TON of traffic that first week.

(Me = me, EB = entitled brat)

Phone rings

Me: Thanks for calling the help desk at <university> this is ITrCool, how can I help?

EB: Yes, I need more bandwidth for my Internet connection in my dorm room. The speeds I'm getting are unacceptable and I can barely connect to the Playstation Network. Please boost my bandwidth to minimum 200 Mb down / 40 Mb up.

Me: I'm sorry, sir, but I do not have control over that. Our network admin and CIO have set and determined the necessary bandwidth allowances for the dormitories along with the other buildings on campus by priority. We don't have enough bandwidth in our pipeline to accommodate that for you.

EB: Well, I understand that, but you see, my father is <CEO of local company that donates to this university> and I think we can reach an understanding that I rightly deserve an extended bandwidth increase due to my family's background. Please stop making excuses and get it done.

Me: Once again, I'm sorry, sir. I cannot accommodate that, nor do I control it.

EB: (in a snooty rich boy tone) That's ok. I'll just go around you and keep escalating until I get what I want. Thanks." click

​

The way we were setup was we had a single bandwidth pipeline for the whole school, and it was split two ways: the main campus buildings and the dorms. The former were "priority" for the bandwidth needs, and the dorms came second. State budgets being the way they were, that's all we could afford financially as our primary ISP (we had a smaller secondary ISP that was just for IT use in the server room if worst-case happened and we had a total ISP failure going on, we could still keep some services going).

So unfortunately, the dorms got less bandwidth and thus less speed. You just had to learn to time out when the best time to connect online for gaming and streaming was, and that was NOT in the evenings when everyone was back from class. For surfing the web or checking email? It was fine. For gaming and streaming video? Not great. But it was what it was. Instead of streaming, you were better off collecting DVD's or renting them from the (then still open) video store in the area and connecting in a LAN with other dorm-mates if you wanted to game multiplayer (or play solo on local campaign mode).

​

I and the CIO and network admin were on casual terms (the IT dept at the school was a tight-knit group), so I later asked them both how that all went down. EB escalated all the way up to the CIO who promptly told him to go pound sand and that he would not be receiving more bandwidth just because he was <local company CEO's> son. If he wanted more bandwidth, he'd have to go get his own personal data plan and hotspot to do that. EB lost his mind and threatened to get his <CEO> dad involved. CIO said go ahead, and called his bluff. Nothing ever happened and as far as I know EB made-due or went and got his daddy to buy him a fancy cellular hotspot for his special Internet needs, AFTER his daddy talked to our CIO and got cleared to do so. (our CIO was a firebrand guy that knew how to handle himself in any situation and knew how to stand up to bullies and read political situations easily).

Unfortunately for us, EB stuck around for a full-ride four years in the dorms and so like clockwork at the beginning of each semester (we reset the NAT restriction filters between semester breaks for security reasons, so you had to get your MAC addresses re-submitted every semester as a result) he would call in with the same snooty attitude about other random junk ("please re-connect the cable jack in my room. The Wi-Fi here is crap." This was after we had disconnected and pulled all old copper connections out of all campus buildings to save on infrastructure cost, save for that which HAD to be physically connected and the brand new WAP's we had mounted covering all rooms of every building). We always drew straws to see who got to deal with him.

TL;DR - entitled kid who had a rich father, thought he'd "escalate to the top to get what he wanted", which was more bandwidth for his dorm room. Ended up losing in the end.

Not a story about myself, but a buddy of mine!
Disclaimer I am not a native English speaker so... c:
Our bosses decided to open a new branch in Switzerland, and a buddy of mine got selected to set up the hardware. Timeframe was a complete horror. You may ask yourself why?
He had two weeks to get a working concept with ordering the hardware, prep it and test

After the hardware was delivered, he went to work, doing a 55-Hour week to set up and test his concept. All went well, and he was sent to the new branch office.

In Switzerland, he went straight to work. He plugged in the NAS and the PSU went POOOFFF. Keep in mind that the NAS is our backbone which hosts a site-to-site VPN, a network drive to share stuff and is our switch between two different internet connections.

After that, he called us and let us know that he needs more time to organize a new PSU/NAS and an electrician needs to look at the situation. The guy who was called measured the voltage of the outlet. The guy told him, "Well 411 Volts, I would guess the NAS is doomed."

After that problem with the outlet got sorted and a new PSU was bought. All went well and I am still surprised that the NAS took the hit like a champ. Still working fine after two years.

TLDR of the story: Even the best prep can go to waste in seconds

We have a branch office which needed a new printer. So they just bought one, after a day or so we get a ticket that they need help with setting up the printer.

Our faces of confusion "They have a printer?"

Well ok lets set it up.

A coworker of mine set it up and the printer couldnt scan and send it via Mail. So he tried for some things and even compared them to the other printer which he set up. After hours of searching for the error and comparing configs. His conclusion: Config is fine, but check still failing (It has a check to see if the printer can reach the user of our mail programm and login)

Another coworker is looking over it, same conclusion.

Enter me, just strolling by getting some water. I saw them just fumbeling and looking confused at the screen. So I entered their office and asked them about the problem and asked them some question. One question was "Did you check the network config?" The awnser was "Yes we did". You may get why I am hinting at it

After some tests I came to the conclusion that the printer cant comunicate to the internet. So I open the network config. Config seems on the first glance to be the same as the other printer which got set up and worked fine. Then I noticed it, you cant see the Gateway in the overview.

Find the place to enter the Gateway IP, Empty > Entered the IP, started the check, working.

"Well I´ll be on my way getting my earned water break" c:

I'm a bit surprised this hasn't been posted already as this effects multiple companies.

Someone with more knowledge can give more specifics but the short story is IBM created a line of servers and also created software to interact with the server. This software looks like it was made in the 70's/80's with no graphic interface at all. We bought both from a software company and while we have had numerous upgrades on the server side of things we have never really been told to keep the software up to date.

Two weeks ago a couple of computers started getting a "licensing" error. In talking with the software company IBM had decided any software below a certain version is a security risk and therefore must be upgraded. Sometime over the past few years IBM recreated the program in Java. In order to upgrade though we needed to update Java on the computer and then install this new Java version. Then we would need to configure the software from scratch.

One would think all the computers would get the error all at once but it's been 2 or 3 a day over the past few weeks. We can't figure it out why that is. Luckily, we only have a small amount of computers so everyone at this point is patched. In talking with our software company other companies have hundreds of computers they need to update. I've been told those companies have given IBM and earful enough for them to extend the date on when they need to upgrade by.

​

Tl; dr - if you hit only dead ends, you’re probably assuming something that is not true.

I support multiple multi-tenant OpenShift Clusters. The various business divisions have their own DevOps staffs; we are corporate and own infrastructure.

Last month, an App team contacted me. They were failing DR testing due to a single microservice. Unfortunately it was the key microservice. In production it was called hundreds of thousands of times a day.

The app did little logging around this microservice. Pretty much “called with these args, got this response”. Calls were timing out. Since it was timing out and the service was clearly running, it must be a bug in OpenShift, right? (Oh, did I drop this \s?)

Strangely, api calls to the service timeout; the healthcheck calls get through. Oh, wait, it’s failing those healthchecks! App team turns up logging for the service and finds the service is deadlocking. It’s technically not dead, but it’s not answering the phone.

I feel vindicated, Apps fixes it, DevOps deploys, and DR passes. Oops, no, it doesn’t. Instead of a timeout, the call fails.

I spent two days tracking the progress of the API call. The api is called with a url containing a fully qualified domain name. Working from the outside in, I rule out DNS^1, the cluster’s BigIP, and the cluster’s ingress routers. Each shows the api call entering and leaving.

That leaves the pod containing the service and the node it is running on. More tcpdump- yes, it reaches the node, no it does not reach the pod. The definitions of the components (Route, Service, Pod) are correct. They’re working perfectly in the production namespace on the primary cluster.

Stumped, I ask a more experienced coworker to look at it. After a few hours he independently verified my results, but was also stumped.

I enter a ticket for Red Hat support. I tell them I’m sure I’m overlooking the obvious and need another set of eyes. (Ooh, look foreshadowing).

The next morning my coworker says, “You’re going to laugh…”. As aside that phrase ranks right up with “We need to talk” on the list of things I hate to hear. On our clusters, namespaces (collections of resources) are isolated from each other. You add network policies to describe the traffic you want to allow. There is a set of cluster-wide default policies that make everything work. On our cluster, those policies are copied into the namespace when it is created.

My coworker found that the default policies had been deleted. The namespace contained only a custom policy permitting the DevOps team’s monitoring traffic. Here’s the gotcha: if the namespace has no network policies, the cluster defaults apply. However, if you have any namespace level policies, only those policies apply. The coworker said that’s why we automatically added them when creating a namespace. Adding them back completely resolved the issue. Healthchecks are generated on the same pod, so they’re not affected by network policy.

Post Mortem: the first day of testing the app team noticed the service did not respond. In the course of investigating, Ida Know accidentally (?) deleted the default network policies. The next day, with the problem unresolved they called us.

1 Like users I lie. It wasn’t DNS.

So long time lurker, finally decided after some choice beverages on a Read Only Week to put this tale to pixels...

Backstory

5 Years ago I was working as a 1 man IT shop for a 6 location automotive group. My desk was located at the home office for the group, with 5 of the locations being within a 5 minute car ride to each. The 6th...was over an hour away. When I took over, it was the wild west. The company was still recovering from a ransomware attack, 10% of the computers were domain joined, 3 different domains between the locations...absolute nightmare. I spent the better part of 3 months getting everything built back how it should be when I found out the firewalls were coming to the end of their warranty. I was able to convince the powers to be to allow me to upgrade the all the network equipment at each site. 5 of the sites went smoothly with little to no problems or complaints. Actually got praise that since I took over everything just "Worked" which you all know coming from both end users and management that's a huge thing. Which is where the last location comes into play....

The Install

When I started the project I decided to do the close ones first incase I missed something, or I was too zealous with my blocking of things. 5 Sites in and no complaints meant it was time for that one location over 1 hour away. I get to the site at 7PM, an hour before they close for the evening and start to get setup. I install the network equipment no issue, everything comes up, and all my tests come back good. Its 9:30PM by this point so I pack up, lock up and leave the place to head home. By the time I finally get home, eat, shower and hit the sack its 11:30 almost Midnight.

The Next Day

I get up early and make sure I'm at the location when their service dept. opens at 7am in case there are issues. There are zero issues and the service writer actually thanks me because his tablet actually works farther than 5 foot from the building and the internet seems faster. I make rounds making sure everyone is able to work, nothing is broken, etc.. and that is when "Joe Salesman" comes in. Joe is an old time used car sales man that is the poster child for the stereo typical used car salesman. Joe sits down at his PC and within a minute is yelling (Yes Yelling!) across the showroom for me to "get my @$$ over to his desk and fix his PC" The Problem? He can't get to any of his Pron, Gambling, or other sites that are now blocked by the new (and fully licensed/configured) firewall. He demands that I turn it all off when I tell him why he can't get to those sites, and when I say no, stomps off to what I hope is, smoke a cig, drink coffee, and complain to the others. With nothing broken, I leave and start driving back the hour+ drive to my desk to clean up and make sure nothing else is on fire before I hopefully cut out early due to the massive amount of hours I've put in.

The Call
I get back to my desk, start to clean up/throw out old boxes when the owner of the auto group calls.

...Now I have stories for DAYS about the things this guy has asked me to do or things he said/number of times I've been berated and cursed out on the phone, fired, and then rehired 30min later...but that's not for this story...

The TLDR I can get from him in-between curse words and incoherent yelling is that the location I had just left (the one 1+ hour drive away) was hard down and nobody could get internet/they were unable to work. Puzzled, I check the firewall from my office and it is Online, but there is almost no EGRESS traffic. Strange. Check the switches, nothing too out of the ordinary...Then I see the logs...Something is handing out 192.168.1.0/24 IPs (their subnet was a Class A in line with the other sites). I find the port doing the dumb, and suddenly everything works. Everything is good for about an hour and suddenly everything is down again. The same MAC address is handing out those IPs again, this time on a different switch port! I shut that port and things calm down again. I think I know what's going on so I grab my bag and start driving back to the location.

The Discovery!

So I get to the location and as I do, BOOM the internet doesn't work again. This time, I get more information. Its only affecting the SALES team. Service, Parts, and Guest WIFI are all working perfectly. (Yay VLANs). I plug into a Port I know is set to the SALES VLAN and BOOM 192.168.0.xxx Address is given. I swap to the port I had set for MGMT and set a static IP so I can log into the switch stack. WELL WHAT DO YOU KNOW, THAT MAC IS ON A DIFFERENT PORT!

I don't block the port this time, I follow it back to the patch, and from there go to the desk that the patch runs to. There sits mr "Joe Salesman" rummaging under the desk, typing on the computer, getting frustrated and slamming the keyboard. I wait for him to get up and go smoke (salesmen there couldn't go more than 30min without one it felt like) and go to his desk. Under the desk I find a Linksys Router with the INTERNET port plugged up to his PC and PORT 1 plugged into the jack in the wall. I disconnect it, take it with me, and reconnect that PC to the wall. within a few minutes the SALES dept. was back online and tell the general manager the problem and the solution.

Upon talking to "Joe Salesman" he finally fessed up to trying to 'get past the firewall' because someone he trusts told him it would work (some kid at Wal-Mart across the street). In the end that salesperson left the dealership 2 weeks later citing me as the sole reason and giving the owner one more thing to yell at me about when he was angry.

​

tldr; After bringing a dealership group into the 21st Century, Salesman didn't like that we blocked his favorite websites and took down the whole network trying to circumvent the web filter.

We provide Google workspace and helpdesk support to our customers. We sent this valued customer (VC) an email that we closed out a ticket.

VC: I received this email that you closed out my ticket, but I never received a resolution.

I go back through the ticket to verify resolution was sent.

Snoo-15335: we sent you this message on <date>. If you did not receive it, please check your spam folder.

VC: ( sends screen shot of spam folder) I never received it.

I check the Gmail logs. Not only was the message delivered, I can see that it was read, marked important, labeled, and archived. I ask VC to check for messages marked "important" and suggest that the problem may be due to a filtering rule or perhaps missed communication with her email delegate.

No response from VC, which is about what I expected.

I work for a large USA ISP. Mostly customer service and tech support. I've recently been tasked with enrolling customers in the ACP benefit. Government benefits for the internet.

I'm talking to the owner of the account, the daughter. Her bill isn't what she expected and it's because the ACP credit was taken away a few months ago. She failed to recertify after 1 year. When the government takes its money back ($30) it DOES NOT give it back. Especially 2 months later.

Both mom and daughter tried yelling or reasoning with me to get my company to pay the money. It's the governments money and the governments program. It's out of our hands. Completely, there's nothing I can do.

Upset with my responses the mother takes the phone from the daughter and begins to force burps into the phone. The daughter exclaims in the background "Mom stop you haven't hung up you're still on the phone!"

The mom replies "I know" and keeps forcing belches into the phone.

I say "wow, how childish" and hung up.

What kind of 60+ year old grandmother acts that way?

So I have only just started on helpdesk like less than a month, the company I work for provides white label services to MSP's as a kind of buffer to take pressure off their 1st Line.

So call today very irate user at company A reoccuring issue of not receiving emails from company B. I mean this user was going off gaskets completely blown the lot. See user is meant to be cc'd into every email from company B that any user from company A recieves, think account manager type.

So off to email filter I go to look for said email all the whole filling in ticket in MSP we are helpings system user mentions this has been going on for months. I click the contact name and lo and behold 40+ unresolved tickets over 3 months with the same problem, anyway get into the filter search users email address not a thing that has been picked up by the filter for this user in weeks and certainly nothing from Company B's domain. Confusion ensues.

"There is no email here for you from that sender" user getting more and more irate "well I know it was sent I was on the phone when it was sent", "Okay so you were cc'd in this who was the main recipient" "User B" So I ask for the email address with spelling, lo and behold there it is sender recipient time everything. "Can you forward it to me?" "As you aren't listed as a recipient not really" face down rubbing my temples at this point I look back up at the screen looking at the delivery info I look down to the cc section and her email is right there well I say her email is right there the username is anyway, I'm greeted with:

To: user.bee@domainxxexx.com Cc: user.ayy@domainxxixx.com

"Okay so I have found the email, I'm going to send you a screenshot to your email and you need to contact company B, for the entire 3months you have been working with them I am going to assume that they have had the contact for you saved with this typo"

Go back through the mail filter switch the search to email from company Bs domain and literally every email for 3 months has this typo in the cc line.

15ish different techs assigned to these various tickets, plus user A themself having these emails forwarded on by colleagues when they didn't receive the cc and no one spots it.

Think I did a good today I'm just starting out but hoping to do many more.

One of the higher-ups at my work is partnering with 2 other companies on a project. I'll refer to them as @domain1 and @domain2. my company will be @ourdomain.

i get a call from the higher-up saying that they are having trouble sending an email to @domain1 and are getting a kickback message.

Me: "ok no problem, can you send me an example of the kickback message?"

Higher-up: looks through emails for a while* "no i can't find it"

Me: "alright well how about i try to send an email myself and see if i get a kickback message. what is the email address you are having issues sending to."

Higher-up: "well if you send it from @ourdomain it will go through just fine. it's when @domain2 tries to send to @domain1 that they receive the kickback message."

Me: "so the issue only lies between @domain1 and @domain2, and @ourdomain is working normally?"

Higher-up: "yes, could this be an issue on our end?"

Me: "no"

Higher up: "ok thank you"

My business partner & I are electricians and have an electrical contractor business. He's in the office and I am more so out in the field ( I have the electric license).

I am also the much more of the IT guy, at whatever poor ass level that may be!! ( I have fumbled around and managed to set up a Ubiquiti network, twice. Second network had ~25 wired & 20 wireless devices with 5 switches in 4 buildings with fiber. I did build a Linux machine. During Covid. But more so, because a customer gave me 2 computer cases. Not sure if computer is "properly" set up or not. Too much humble(dumbass) brag?)

At some point his printer was getting cantankerous, sooooo, new printer.

He couldn't get the new one to work. He fumbled around a bit. The wife , on a wired connection hasn't any issues. The part time guy, on a wireless connection, hasn't any issues. Partner, he can't get connection. He asked me to look at it. While kneeling on the floor beside his chair, I wasn't having a good go at it.

A week or 2 go by. He calls HP. They remote in. He says that after 2 or 3 hours, HP couldn't figure out what was wrong.

The 30 days is running out on the return. He asks me to look at it again.

I look at / fumble trying everthing I know, which isn't enough. I finally open a Word file. I tell it to print .

The...... very.....first prompt / error message:

"This is not the default printer. Would like to make it the default printer?"

Yes.

Problem fixed.

HP couldn't figure this out? I just a "dumb electrician.

TL:DR. Pause & Read the error messages , dummy!

PS:

He told me today that he can't / has issues with scanning. "Wife come scan this & email it to me". I want to slap him. The printer sits right beside his desk!!

About 5 or so years ago, our whole company (230,00 employees) was on Lotus Notes for email. I was a Notes Admin and Local Desktop Support who worked with a decent sized group of about 15 who supported a service area of about 10,000. This one year, we received an email celebrating Black History month, with some details about an event. I believe it went out to all NCAL users, I think it was around 40,000 employees who received the event email.

In my group, we had a diverse set of people from all over the world. We had one guy from Sudan, let's call him Ma'bad. He was starting his family, already had one kid with one in the oven. I liked him, although we were from very different cultures, I had respect because of the stories I heard about Sudan.

Back to the email. There was this option in Lotus Notes to add an email to a new Calendar entry. This is exactly what Ma'bad decided to do. Now, there were two options, one to add it to your Calendar, and one to add all recipients to your new Calendar event. Ma'bad either by mistake or not looking, choose to add everyone to his event, this included all 40,000 people. At some point, he realized what he did when everyone started asking him about the event. And in a panic, he canceled the meeting. This triggered a canceled meeting notification to all recipients in the event, so everyone received an email stating "Black History Month with Ma'bad has been canceled". Then in a secondary panic, he pulled in a senior tech and requested "Oh snap support" to assist him in his plight. He was then considering a recall, but by then, the servers were locked up and the recall feature was only for emails crafted, not automated cancelation emails.

I felt so bad for him, and then he soon left to a different department.

So I've recently come from a big project back to the service desk. Taken a week or so to get back into the flow of things again, answering the phones, getting to grips with minor frequent issues and such.

Little did I expect one of the requests I had dealt with today .... SMH.

A user had logged a request following extended leave from their work, for the most part we would replace the device for security reasons as the laptop is more often than not disabled due to being out of corporate policy.

The end user was using the laptop regularly enough for this not to happen. However following a major incident the user wasn't able to log in, prior and following the incident.

The request had been logged to the service desk "triaged" and not solved (multiple password resets). No-one could "solve" this request so it was passed to the escalation team (before I joined) and they couldn't solve it.

Escalation pass it to our end user compute team, they still can not figure it out, why this user despite having by this point well into the double digits password resets.

End user compute see fit to pass this to starters and leavers (a reasonable move) to investigate any account issues

To keep this post readable this ping pong of the request goes on for a MONTH a literal month.

I picked it up this morning, looked at the back log and scratched my head for a few mins, called the user and got the run down.

I reset the users password again ( they had forgotten what it was last set to and I wanted to be throurough)

The user tried the password and low and behold it didn't work.... unsurprisingly.

So I did the one thing a dozen other people before me didn't I asked the user to check their WiFi connection.......

Queue internal rage .....

The user wasn't even connected to their own WiFi!!!!!

I asked them to try their new password once more and guess what ! It worked !!!

TLDR; Rule 101 of tech support, make sure its connected !

In this circumstance, it was. And, it was my fault yet it was no fault of mine.

Recently made some changes to the server and storage infrastructure, going from an unstable vSAN solution to a proper SAN. Racked up the SAN at the datacenter, then confirmed the networking configuration on the 10Gb switches. Configured the two HBAs with the necessary IP addressing and then hooked it up to the network.

Interestingly enough, 2 of the hypervisors are able to see everything on the network, and 2 hypervisors are not able to. When all 4 were connected to the network switch or the HBAs none of them could communicate over the network

Digging into what could be going on with the networking found a very interesting thing. The two 10 Gb switches were had the exact same configuration on them, but one switch was not allowing any traffic to flow through them. And, even more interesting, were configured to be stacked but were not stacked.

First decided to look at the stacking config and debug, exept neither switch was trying to communicate as a stack. One switch was also refusing to show the current or saved configurations or connected SFP+ transceivers. Removed the stacking configuration and re-applied, with and without reloading, which seemed to make no difference. Decided to just remove it.

After a bunch of trial and error, confirmed that once the two switches were disconnected from each other and both not connected to the same HBA there was network communication. You are thinking about spanning tree, and it being the culprit. Oh boy, are you and I wrong. Changed the spanning tree configuration on the hosts and switches, nothing seemed to help. Network completely died when thr switches were directly or indirectly connected. But, still must be STP.

Added a new VLAN, configured switch up for PVSTP. Nothing on that VLAN could communicate, and this is where I learned that even though it is assigned to an interface it doesn't actually work unless it is assigned as the main and inly VLAN on that interface.

Went ahead with assigning the VLAN to an unconnected port for testing, and started to reach for a new SFP+ transceiver. As I am standing back up after digging through my bag all I saw were error lights on the networking and servers. Only, these errors were from the switches that handed the data side and not storage. These switches are not connected to the storage switches in any shape or form. Waited a bunch, then decided to power cycle everything, and this is where bad went to worse.

The storage was already in production, as we had to move our guests from the vSAN to the proper SAN due to a DR scenario. But, the SAN was working properly with 2 hosts which had working redundant links through the storage switches to the SAN. The SAN lost all network connectivity, meaning no guests were coming up. No DNS, directory services, client and internal apps. Nothing.

Reverted that single change, the port that still is not connected to anything. Removed the storage switches completely and went direct connections from the dual-homed NICs to the SAN. None of these changes worked. Frustration settled in, and while I am normally the coolest person in the room when everything was on fire I was not the coolest in the datacenter.

Minutes changed to hours while troubleshooting. When after calling a few people to spitball ideas at 4 in the morning, it hit me. The 2 hosts with a lot of issues were still connected to the storage switches, which were still connected to the SAN HBAs. Disconnected the DAC on both, leaving the fiber transceivers and reinitialized thr iSCSI connections and it all came up.

Spoke with the switch vendor, they are unable to figure out why that network change broke everything. Could be a coincidence, but after going bacn to the datacenter today I do not think so. Both switches have redundant power supplies. Both on one of the switches report an error when power is plugged in, though unable to get an error code. The other switch had only a single working power supply.

The support rep from the manufacturer did not see these issues in any of the logging. These switches were also managed over the network during the call last week, while one wasn't this week.

Fortunately, the only people not receiving their pizzas were the poor folks that may have tried to order pizza at 4am.

UPDATE:

Was able to nuke the switches from orbit, loading clean firmware on a clean filesystem. Did some additional packet captures for diagnosis, saw packets leaving the NICs and coming back in, but Windows was not seeing the packets coming back in. After digging into the drivers today found the server had a tonne of QLogic and other emulation drivers for iSCSI, iSNS, FC, and other protocol. After removing all these emulation drivers, then removing and reinstalling the NICs through the device manager, all networking communications worked. Too many drivers were trying to read the incoming packets and just not handing the packets back to the Windows kernel for further handling...

I once worked, under contract, to help a company set up an IT department. Like a lot of other companies that existed before computers did, each department bought its own computers, and some set up LANs, while others set up peer-to-peer stuff, and others used sneakernet. This company brought in consultants to tell them they needed an integrated IT department & network, and the company hired a CIO who was a damn good one except for when he hired his idiot son because his demon seed couldn't hold down a job, not even as a used car salesman. (I met his other sons. Nice guys--they ran a pretty successful plumbing business.)

The CIO started building his IT department and when our contract expired, he let everyone from my company go except for me, because (humblebrag) I'm damn good at fixing hardware. I was still under contract through my old company, though. Important to this story: This was back in the DOS & Windows 3.1 / Netware days. Also, even though the CIO's numbskull son could barely breathe and walk at the same time, this dumbass treated me like a second-rate citizen. Honestly, a dirty dinner plate would have done a better job than this jackass.

The CIO was the only person with a laptop, and one day he gave it to his meathead son to fix because Windows stopped launching. This ignoramus had no clue what to do, so he gave it to me.

Hardware checked out. SCANDISK didn't show anything alarming. A few other tests told me nothing. I started scrolling through configuration files, just to see if I could get another idea.

Now, DOS and WIN 3.1 used a lot of configuration files. You might have heard of the DOS config.sys and autoexec.bat files; they set up the computer by loading drivers and any programs that might be needed down the road. After that you launched Windows--either manually by a command-line command or by putting that command in your autoexec.

Windows also had a bunch of enormous text-based configuration files that loaded stuff for Windows and made it work, and for the life of me I canNOT remember what those files were called, or even their extensions. But I was scrolling through one of them just to buy time while hoping another idea would come up.

So I'm scrolling and scrolling, and I'm not sure my eyes were really focused on what was going past them but suddenly . . . whoa. Scroll back, and I put my cursor just . . . there.

There, where there were two spaces where there should only be one. I removed one of the spaces, saved the file, and damned if Windows didn't launch and purr like a kitten. It must have shown on my face, because bonehead said something to me, no doubt with a sneer. I turned the laptop toward him, with Windows up and running

"I'll take it to my dad," dipshit told me.

"No, you won't," I answered and I took it to the CIO. He asked me, first, why I brought it to him, and I told him. He shook his head ruefully, then asked me how I fixed it. I told him that, too.

"You're kidding me," he replied, reasonably, IMO.

I just answered, "Nope," and asked if he had anything else before I left.

I'd love to know how that second space got there.

​

My gf is a bookkeeper, perfectly capable of using technology as far as an end user is intended to be able to, but by no means is she interested in being the ‘IT person’. Yet, being that shes still the most technologically adept person in her office, thats obligatorily one of the many hats shes forced to wear.

So her boss called her into his office yesterday cuz his numlock key popped off and he didnt know how to put it back on……….. just gonna let that sit for a sec………..

Anyway, she popped it back on for him, prob with a pretty snarky face.

Today she went into his office and saw a bottle or something strategically sitting on his numlock key. Naturally, she inquired as to why thats on there so conspicuously. And his reply:

“Yeah, that key just keeps popping off so i glued it”

😂🤣😂🤣😂🤣😂🤣 dude im dyin 😂🤣😂🤣😂🤣

Me: Me

O: Owner

My first job was editing videos and photos in a photo-studio with the cheapest owner ever. I was paid $4 USD a day, but I accepted it as an opportunity to learn photoshop.

Anyways, O was always cutting costs and this bit him in the ass when one of the HDD died in a PC (he had 3) with some projects he hadn't delivered yet. That day I talked to O about buying a NAS and having a Raid 1. He accepted having a NAS, but of a single bay because he wanted the cheapest option avilable.

I stopped working there soon after and didn't have contact with O.

Cue two months ago. I'm called because the 8TB NAS died and O was having a crisis because his whole business was there. I told him his only option was sending the HDD to a laboratory, but he received the news that nothing was salvageable.

I talked to him and convinced him of buying a NAS for a raid 1 and he said that yes, it was necessary. So I researched about NAS and found an excellent option and sent O the link. He said that he found another one that was the cheapest available and he would buy that along with the cheapest HDDs.

I'll be waiting the call in the next years when the NAS burns his HDDs or something. Talk about learning from our mistakes...

​