/r/sysadmin

Photograph via snooOG

A reddit dedicated to the profession of Computer System Administration.

A reddit dedicated to the profession of Computer System Administration


Rules

  1. Community members shall conduct themselves with professionalism.

  2. Do not expressly advertise products or services outside of approved threads.

More details on the rules may be found in the wiki.


For IT career related questions, please visit /r/ITCareerQuestions


Please check out our Frequently Asked Questions, which includes lists of subreddits, webpages, books, and other articles of interest that every sysadmin should read!

Checkout the Wiki Users are encouraged to contribute to and grow our Wiki.

So you want to be a sysadmin? RTFM


Sysadmin Jobs

Official IRC Channel - #reddit-sysadmin on irc.libera.chat Official Discord - https://discord.gg/sysadmin


/r/sysadmin

974,454 Subscribers

5

For those of you dealing with PCI compliance you should know that SAQ A just changed 60 days ahead of the March 31st deadline

The gist:

After thorough consideration and review of industry stakeholder feedback, PCI SSC is making the following updates to SAQ A:

  • Removal of PCI DSS Requirements 6.4.3 and 11.6.1 for payment page security, and Requirement 12.3.1 for a Targeted Risk Analysis to support Requirement 11.6.1.
  • Addition of an Eligibility Criteria for merchants to “confirm their site is not susceptible to attacks from scripts that could affect the merchant’s e-commerce system(s).”

https://blog.pcisecuritystandards.org/important-updates-announced-for-merchants-validating-to-self-assessment-questionnaire-a

2 requirements removed as long as you can attest that your website is not susceptible to cross-site scripting attacks - which - in order to attest that you have to have the monitoring and controls on scripts in place anyway

See here:

https://sourcedefense.com/resources/blog/assessing-the-new-saq-a-changes-insights-for-qsas/

0 Comments
2025/02/01
19:49 UTC

1

Large facility build out planning, looking for advice.

First time I've ever been tasked with planning, installing and setting up a 40K sqft warehouse with offices.

I've installed/wired some places before, but nothing larger than 10-12K sqft.

I'm doing a site visit next week. I should get a copy of the floor plans while there so I can upload them into Unifi's site builder.

I'm guessing I'll need to setup and IDF on the far side of the facility connected back to the main using fiber.

I've done swap outs in a facility this large, but never 100% on my own start to finish.

Any tips/tricks/advice? Guestimating 15-25 cameras as well as a full network.

0 Comments
2025/02/01
19:41 UTC

0

How does Group Policy User Config know which User to apply to ?

I'm sure this is likely a really dumb question but I can't seem to find a clear answer.

I have minimal experience with Group Policy (last 10 years or so has largely been MDM (mobile device management) in the cloud with Apple and Android.. so I don't deal with Group Policy much at all. )

I understand there are 2 parts to a Group Policy

  • Computer Configuration

  • User Configuration

It makes sense to me that the "Computer Configuration" applies to the entire computer (as long as the Computer Object in AD is moved into the same location as the Group Policy)

But how does User Configuration work ?.. there's a particular Group Policy I've been editing and testing. I see it applying correctly to the AD User account I'm logging in as,. but that AD User Object is not in the Group Policy OU with the 4 Computer accounts. It's in an entirely different AD OU.. so how does the Group Policy know it applies to that 1 specific User ? (or does it apply to any Domain User that logs onto those 4 machines ?

Also.. does "User Configuration" NOT apply to Admin accounts ? (it doesn't seem to when I test)

8 Comments
2025/02/01
19:24 UTC

1

Audit Detailed File Share tips

I briefly explain why I want to enable this audit policy:

It seems that last week someone or something recursively deleted about 150 - 300 internal directories from some shared resources. Apparently they are rarely used and I received the incident last week, but they were deleted a month ago.

Well, I had no choice but to use snapshots and compare the changes, to know which directories were deleted, a disaster and a tedious job.

It occurred to me to enable this policy to see what changes are made to the resources but the logs they generate are too many and not very understandable, especially when you change the name of a directory and it is interpreted as a "fake delete"

This is the only way that Windows allows you to audit these changes, as far as I have seen. How could I manage the huge creation of logs? Since I would have to compress them to have some history. Is there any script to detect mass deletions of internal directories of resources?

It seems to me to be a completely serious mistake, from my point of view, that my department does not control these things, that such quantities of directories or resources disappear and are not detected after a month, and due to a user's notice since he could not access them.

Is there any third-party tool better than the audit? Any advice on how to manage all this in the best possible way?

Thanks.

0 Comments
2025/02/01
19:04 UTC

0

How do companies let you know that you got the job?

Is it through phone calls or email? Almost every job I’ve had involved me working through staffing agencies who called me to tell me that their client wants to extend an offer to me

So how do companies do it directly?

6 Comments
2025/02/01
18:55 UTC

14

What's something you accomplished this week?

In light of all the rant threads we see, what success have you had this week?

New job? Automated something? Project Complete? Cool new hardware?

50 Comments
2025/02/01
18:47 UTC

1

VPN for Remote Office

Hello all,

Looking for advice on setting up connectivity for remote office and would appreciate your input on the cost effective solution.

10 employees connected with company owned desktop PCs having Fiber Home Internet with ISP router

Requirements:

Employees need secure access to applications hosted on our on-premises data center.

Devices must be AD joined with DCs resides on-prem

We want internet access to be limited and filtered. But, require local breakout to avoid all traffic backhauling through the data center.

Endpoints should connect to the CrowdStrike cloud for protection.

Given these requirements, what would be the best, yet cost-efficient, approach?

Would a UTM firewall solution security features and build VPN tunnels?

Is there any other better alternatives in 2025 for such use case

8 Comments
2025/02/01
18:35 UTC

7

Architectural firm sharing 25TB with multiple offices internationally

How would you set up file sharing of 25TB for 200 users across 5 offices internationally with about a dozen or so strictly remote workers? Each server would have some data only needed for that office and some that would be shared across. It's a mix of lots of small documents (Office, PDF, etc), with larger CAD/Revit and analysis files as well. OneDrive has been used on each server to sync across to other servers as we're on the M365 platform and while I know that's not a great choice at all and should be swapped with a DFS setup, it's worked surprisingly well.

In a current setup with local Windows file servers at each location, LAN users are happy but some remote workers and traveling laptop users complain about VPN being cumbersome in accessing SMB shares. How would you propose improving this situation, even if it's a complete infrastructure rework (and implementation budget weren't a main driving factor)? Maintenance budget is more of a concern though as IT staff is small.

Any help would be appreciated!

28 Comments
2025/02/01
18:14 UTC

24

Anyone else suffer from "imposter syndrome"?

I spent 15 years in multiple IT roles with a very large auto insurer. I was mainly on the Performance and testing side of things, Network Performance Analyst, Infrastructure Analyst and a stint as a Data Analyst.

I never graduated from college, just 2 year Associates Degree but was lucky to have been hired in as a entry Network Analyst and learned so much over those 15 years.

I was laid off from that job 5 years ago and ran my own 3D printing farm for a few years and about 4 months ago I took on a job as an IT Lead at a very small company, like 20 employees.

This place has been around for 40 years and their IT is a cobbled together mess of older refurbed hardware (they are very cheap)

I am struggling trying to get a grasp around the nightmare network they have setup and issues that are coming up.

There is next to no documentation for the hardware, the patch panels and switches aren't labeled, runs of cabling are zip tied between buildings it is just a mess.

One of the buildings has lost all network connectivity, I ordered a ethernet tester and probe to try to test the runs and figure out where everything terminates at. And to top it off the WiFi went out on Friday at the end of the day and I can't even find the key to get into the server cabinet that has the FortiNet firewall that the Linksys wifi router is connected into.

Sorry for venting and feeling inadequate

39 Comments
2025/02/01
17:53 UTC

0

Barclays IT Outage

Any suggestions or ideas what's caused this ?

2 Comments
2025/02/01
17:45 UTC

1

Two RD Web Access servers - conflict? HTTP 500, duplicate collection

Hi guys,
I've been dealing with some issues on one of our RDS farms. There are 5 servers in total - gateway, broker, web access host, and 2 session hosts, all Server 2019s. As for the networking side, our FW has a public IP and we use a VIP to redirect traffic for the RD website.
I wanted to install RD Web Client feature to make HTML5 client available for our customers so they wouldn't have to download any RDP files. However, after installing the WebClient package, I couldn't get it working as the IIS throws HTTP 500 error and keeps complaining about duplicate collection:
https://postimg.cc/bG1603d7

This is my web.config file content:
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<system.webServer>
<defaultDocument>
<files>
<clear/>
<add value="index.html" />
</files>
</defaultDocument>
<staticContent>
<mimeMap fileExtension=".cer" mimeType="application/pkix-cert" />
</staticContent>
</system.webServer>
<location path="index.html">
<system.webServer>
<staticContent>
<clientCache cacheControlMode="DisableCache" />
</staticContent>
<httpProtocol>
<customHeaders>
<add name="X-Frame-Options" value="SAMEORIGIN"/>
</customHeaders>
</httpProtocol>
</system.webServer>
</location>
</configuration>

When I remove the <customHeaders> section, the webpage loads but is almost completely empty, without any remote apps, only the top bar is present.
https://postimg.cc/SXLrYRvq

I'm starting to think there must be some conflict between the gateway and web access hosts as both have the RD Web Access role installed - the people who configured the farm like this, already left our company - I assume they probably wanted HA for the web access. I also checked the VIP/real server settings and the traffic from the firewall is forwarded only to the gateway host IP, no redirection to the second web access server.
Do you have any ideas what should I check or reconfigure to get this working?

0 Comments
2025/02/01
17:40 UTC

0

Looking for the Best Dell Precision & ThinkPad for High-End Development

I'm looking to buy a high-end Dell Precision or Lenovo ThinkPad for development and model training (AI/ML, heavy workloads, etc.). Performance is the top priority, so I'm fine with workstation-level specs.

Thanks for your suggestions :D

24 Comments
2025/02/01
16:36 UTC

20

Why does SNMP keep turning itself back on random HP printers?

We have monthly vulnerability scans on our network and each month we’ll see a different, random HP printer come up with the vulnerability “Default SNMP Credentials”. We will fix these each month either by setting a unique SNMP string or disabling it altogether. We’ll rescan the device and verify the vulnerability is cleared.

Then the next month another random HP printer or two will show up in the new scan with Default SNMP credentials. What keeps randomly turning SNMP back on with the default string? We keep turning them off but then each month another random one is back on. It’s a never ending game of wack-a-mole. How can we prevent SNMP from changing its setting on its own?

This happens on all models, from little HP 400’s up to the big MFP’s like Enterprise Flow M631.

18 Comments
2025/02/01
16:20 UTC

2

Firewall Swap Help

Hello, I am looking for some help with a network deployment that I am a bit over my skis on. I am a jack of all trades but a master of none and this one has me stumped. In a managed switch environment with multiple VLANs I would create the VLANs on the switch and firewall and have the firewall as the gateway on each of those VLANs. In an environment that I took over the managed switch is the gateway. I have never administered a network like this. I am in the process of swapping out a Cisco ASA for a Fortigate 90G. Here is a breakdown of the setup and where I am stuck.

There are about a dozen VLANs on the switch but for simplicity's sake let’s just focus on 2. VLAN 100 is 192.168.100.0/24 and this is where the client devices and servers live. VLAN 150 is 192.168.150.0/24 and is where the gateway sits. The gateway on VLAN 100 is 192.168.100.1 which is the IP of the Aruba switch. The IP of the Cisco is 192.168.150.254. I setup the LAN interface of the Fortigate with an IP 192.168.150.251. If I connect directly to this interface I can get out to the internet, so my policies and routes are good in that aspect.

When I plugged the Fortigate into a port assigned untagged VLAN 150 I could not ping it from VLAN100. I reviewed the Cisco and found some route commands and after entering this route into the Fortigate I was able to ping the Fortigate from any device on VLAN100

Route 192.168.100.0 255.255.255.0 192.168.150.1 (the IP of the Aruba on VLAN150).

I thought I was almost home but no. On the Aruba here is the route out command.

ip route 0.0.0.0 0.0.0.0 192.168.150.254

So I grabbed a test device on VLAN100 and create this additional route in the Aruba.

Ip route 192.168.100.21 255.255.255.255 192.168.150.251

I immediately lost internet access on that device.

Here is where I am stumped. I am assuming I am missing some additional policy or route on the Fortigate. My current policy is an ANY ANY from that LAN to WAN.

My goal is to route VLAN 100 out via the FG to make sure everything is working correctly before I start routing all VLANs out via the FG.

Any help is appreciated.

6 Comments
2025/02/01
15:57 UTC

12

Windows Update unavailable?

Windows update is unavailable, Autopilot deployment is failing, and installing apps (Company Portal) from Autopilot and the Microsoft Store isn’t working.

https://update.microsoft.com

Certificate error and a 404.

8 Comments
2025/02/01
15:30 UTC

0

Winget fuzzy search?

Hello! I found fzwinget for fuzzy searches but I can't get it to work. What do you guys use? Thanks!

4 Comments
2025/02/01
13:48 UTC

303

Caps lock instead of shift keys?

Do any of you old-timers notice that the new kids being hired turn on the caps lock, type a capital letter, and then turn off the caps lock instead of using the shift key?

313 Comments
2025/02/01
13:26 UTC

4

Freshservice and Ticket Replies not arriving in Freshservice

We've been on FreshService for about a year. Been fine. Mailbox is 365.

Recently, some customer replies to a ticket are not reaching FreshService. We changed the setting on the 365 Mailbox connection to keep the mail in the Mailbox when this first started to happen. The reply is clearly in the mailbox, but just never reaches Freshservice.

It's not in FreshService spam either. Their support is not helpful at all. blaming Microsoft, but it cant be a MS problem as the email is sitting in the inbox.

Anyone else encountered is issue?

3 Comments
2025/02/01
11:31 UTC

44

What are your IT related conspiracy theories (just for fun).

Mine:

When a compromise occurs it’s a sign that god is angry.

Building a PC is made difficult purposefully by the manufacturers in order to haze PC gamers into an international clan (ow I cut myself!).

DeepSeek is a secret plot to undermine American confidence by attempting to make fun of English speech patterns (it keeps saying Wait! As its thinking every paragraph 🤔🤨)

What are your IT related conspiracy theories?

142 Comments
2025/02/01
11:29 UTC

0

WiFi not working good

hello

I need your help. I've started a new job and the manager of an external location came to me and told me that the network isn't working well.

The first point is the WiFi. It's a 4-story building and has 3 Aruba access points installed on each floor. Aruba no longer has a controller on site but is managed in the cloud. I don't know the size per floor yet.

According to her, you keep losing the connection , the signal strength varies and its slow.

She also told me that all smartphones and tablets that are only connected to the WiFi have the wrong time?

I would like to go there and take a look and measure it. Do you have good (not too expensive) software for WiFi checks? I want also try Iperf, but can i also use them for WAN connections?

Do you have any other advice?

Thank you :D

4 Comments
2025/02/01
10:45 UTC

4

Recommendations for M365 and related trainings/certifications

I am currently a SysAdmin for a small company with about 30 users - mostly on-site using AD, but with a handful of remote users. The company also has an M365 subscription which has primarily been for email up to this point. Most users have an M365 Business Basic subscription, but we will be switching to Business Standard or Business Premium in the near future.

I've been with the company for 3 years, and after fumbling around for a while I have a pretty decent understanding of how to administer our M365 account now. Managing users (through M365, not Entra), licenses, SharePoint, etc. As well as Defender for M365. (I was hired primarily help desk purposes and Web site administraton)

With that being said, there is a desire to start taking advantage of more of the M365 services, specifically user and device management. We may be adding several remote offices in the near future, so there will be a greater need for these services. We currently use Apple Business Manager for phones and nothing for laptops.

I will be attending a boot camp soon for M365 Endpoint Admin and (MCA) and M365 Administrator Expert (MCE).

I am the entire IT Department for the company, plus we have a contractor who comes in once a week to take care of the on-site server maintenance. The contractor is mainly a server guy for small businesses without much background in M365.

I'm looking for some feedback on specific trainings/certifications I should consider for our environment. Specifically to get up to speed with Entra ID, Azure, and InTune. I'm not necessarily looking to get Microsoft certifications, but wouldn't mind following the certification path for more structured training.

2 Comments
2025/02/01
09:36 UTC

10

Allowing local admin rights on demand?

We have a need for some end users to have local admin rights some of the time, but of course not all of the time.

It's for a variety of reasons but usually it comes down to needing to change IP details or add/remove/run software some of which is really poorly written and insists on having admin rights and there is enough of it that figuring out exactly what rights are needed isn't always practical plus the official vendor position is "you need admin rights".

Other than providing second accounts that can be used to elevate what are you using to give temporary admin rights when people need them please?

All Windows 10 and 11.

Jas

37 Comments
2025/02/01
09:03 UTC

0

crear script para evitar que un usuario de dominio se inicie sesion en 2 o mas equipos al mismo tiempo

Hola buenas noches, no se de ingles por ello escribo en español espero me puedan ayudar.

tengo un inconveniente con los trabajadores de la empresa que yo les creo usuarios moviles para que inicien sesion en los computadores para que cumplan con sus funciones. Lo que pasa es que los nuevos empleados que llegan no se acercan a la creación de usuario y los mas antiguos estan emprestando sus usuarios y en ocasiones en 2 equipos o hasta mas equipos abren el mismo usuario al mismo tiempo o simultáneamente, esto a acarreado un problema grande y no he podido solucionar.

alguien sabe de alguna GPO o que configuración hacer para solucionar este inconveniente? como Crear un script para evitar que un usuario de dominio se inicie sesión en 2 o mas equipos simultáneamente.

Espero me puedan ayudar.

1 Comment
2025/02/01
05:34 UTC

1

Question: MS ADCS AutoEnroll to only a certain OU

Premise:

I want all of my servers in a specific OU to auto enroll for a Cert Template called "RDPAuth". Currently, I have a security group called "PKI-RDPAuth-AutoEnroll" in AD, and I've given it AutoEnroll permissions to that Template. If I manually add a server to that group, then it will enroll for that template on the next reboot. Cool, fine... that works.

Problem:

I don't want to have to remember to add every new server to that security group manually, and in some VDI setups automatically spin up new server automagically as needed. There's gotta be a way to dynamically add/remove computers in an OU to a security group.

Gotcha

the final bit: I don't want too run a scheduled task daily, that runs a script, that enumerates the OU computers, and adds them too the group. This is just to janky and there has to be a cleaner way to implement. Maybe via GPO some how?

Eventually I would like to do a similar setup with User certificates, and I don't want all of my service accounts requesting one too... plus I think HR is a revolving door some days

8 Comments
2025/02/01
04:34 UTC

1

Finding out who has access to "msFVE-RecoveryInformation objects" property

I'm trying to figure out who all has the ability to read the AD property "msFVE-RecoveryInformation objects" aka the Bitlocker Recovery Key. I know 'Domain Admin' group by default has access, but I can't figure out who else has access. Our Help Desk team has access, BUT none of the groups that they are is a member of, would have been delegated access.

I've done google searches, but all I am finding is HOW to delegate access, but nothing about how to audit the access.

Any help/idea?

3 Comments
2025/02/01
03:28 UTC

1

Hybrid AD/365 - Leaver account processing (OneDrive)

Hey team !

Hoping I could please ask for some insight into the different processes different people follow with regards to leaver accounts / data retention within 365.

Here it is an on-prem / cloud hybrid model. Users + Devices are on prem in AD syncing up to Azure (No intune or AzureAD/Entra configuration). Exchange + OneDrive + Teams + SharePoint all active and cloud only.

Currently the process when offboarding is to: 1) disable user account in AD + 365, 2) convert to shared mailbox, 3) remove 365 license from user account.

I would prefer to not have the tenancy clogged up with endless leaver accounts in a disabled state that are sat AD syncing up to Azure, however I am cautious to move these objects into an OU not syncing to Azure as this would set the OneDrive's into 30 deleting countdown mode.

I am exploring the option of creating a single service account which holds all ownership of ex-employee OneDrive's to enable the ability to delete the original user account. Alternatively maybe manually saving the data into a Leaver's Data SharePoint site ? Please advise.

How do you all handle your offboarding in 365, whether hybrid or cloud only.

Would love to hear your input as I am looking to do as much as I can to clean up this tenancy prior to migrating into a fully cloud based environment using Entra/AzureAD + Intune so the on-prem infrastructure can be decommissioned.

Cheers !

0 Comments
2025/01/31
04:01 UTC

3

Windows Users GPOs get applied, but not Computer GPO

Have created a Test OU, where I'm testing GPOs before pushing them to the Domain. We are using AWS WorkSpaces only. The WorkSpace I'm testing against is successfully joined to the Domain.

I am able to successfully apply User GPOs, but not Computer GPOs. Can prove this from the WorkSpace by running gpresult /r from the Command Line.

Some troubleshooting:

- The GPO (in this case is Google Chrome) shows Link Enabled

- Under Scope tab, the OU shows up

- Security Filtering shows Domain Computers and Authenticated Users listed

- If clicking on the OU from Group Policy Management, under Linked Group Policy Objects Tab, I see that Google Chome shows Link Enabled set to yes, and GPO Status set to Enabled.

- From the WorkSpace, checking Event Viewer and Applications and Service Logs, don't show any errors for Group Policy

Can anyone provide other things to look at or other things to troubleshoot?

24 Comments
2025/02/01
02:56 UTC

8

If you were allowed a fresh start with your IT system, what would you do differently?

Hi all,

I'm currently faced with the task of migrating a client with an absolute mess of an IT system, with different programs and logins used for a variety of business functions, into a more streamlined and secure approach. The previous MSP that ran this client's IT system did not make use of any group policy, or integration into M365. Additonally, they were quite lazy when it came to security, such as terminal server access.

So as a result, I'm using this as a fresh start to get their IT system working in a way that will prevent headaches for my team in the future, as opposed to having a whole bunch of policies and systems that will be "too difficult" and "too disruptive" to undo in the future.

The client is wanting to use Microsoft 365 services as much as possible (think SharePoint and Teams). They will be hybrid joined (non-negotiable apparently), but I'm looking to make full use of InTune, which is something that I want to get right.

So basically, I'm wanting to know, if you had a generic IT system that had around 150 users, which was looking to expand in the future to at least 500 users, what would you do?

Conditional access, Intune, printer setups, GPO, AutoPilot. All that jazz. If you were able to start afresh, what would you do?

40 Comments
2025/02/01
02:50 UTC

30

Storytime: Ransomware, worst case scenario (~2015?)

This took place before the age of double ransom - exfil and lockout. Some of you may recognize this story - if so, hi guys! And I apologize, this is a bit of a long one, but hopefully enjoyable in its sad cringey sort of way.

So a number of years ago I worked for a quite large NFP, I think we had probably around a thousand users at the time. Well, they used to be an NFP, not sure if they still are, at least fully. They got tired of selling all the Koolaid and started drinking it themselves, which is to say the execs wanted a lot more of the pie.

One morning I walk in and sit down at my computer and before I can log in the junior sysadmin says "Hey, we got hit by ransomware" and just starts laughing. He was a funny guy, and I couldn't always tell when he meant to be funny. I knew he was serious though. To myself I was like "Well I don't need coffee now," and logged in to check for myself - maybe preserve my device. Lo and behold, those lovely little text files we all know and love sitting all over my desktop.

I confirmed I was seeing it too, and he laughs again and goes "Yeah I'm watching files be encrypted realtime on the server. You should see this." Walk over and take a look and sure enough one by one they're being encrypted, and just then the sysadmin walks in. He goes to his desk and the junior says "Hey, we've got ransomware." Without seeing anything he just goes "No we don't."

I go "Yup, I just confirmed it, mine is locked too."

He just snarkily goes "Well maybe YOU have ransomware," as if to accuse me. I asked if he'd even taken a look knowing full well he didn't. He logs into his system and as it turned out HE had it too, something he was more than a little grumpy about. He looked over the shoulder on the junior and just rolls his eyes. After a few minutes the manager comes in and sees what's going on.

The manager himself was not a terrible sysadmin but he obviously didn't know a thing about ransomware, despite reading all day and watching lots of tech YouTube. He's throwing out completely absurd and unhelpful suggestions and I'm just shooting them down left and right. "Pull the plug on the servers, now," I tell them. "If we haven't taken stock of what's infected, it's clearly spreading across the network and we need to save what we can, or at least stop the spread."

The sysadmin said something to handwave the suggestion saying we still didn't know what was infected, and one of the helpdesk said there were a bunch of tickets about ransomware from basically every site.

Just about then our fearless CTO walked in to say his usual (occasional) good morning. His demeanor said nothing was wrong to him, meaning he'd just gotten in as always. "We have ransomware throughout the network, including servers," I told him to head off the denials. Apparently I hadn't headed them off. "We don't know that" the sysadmin said, still apparently doing nothing. "Shut it all down" is all I can say.

We showed him the files being encrypted, and the example of my desktop. He went to the sysadmin and just talked with him, who continued to deny everything. The manager went over to talk with them, again offering suggestions not based in reality. "I found a product that can reverse ransomware!" he said triumphantly. This was before some of the ransomware had started having decryptors published and other resolutions. Our AV had no idea what to do with it - Symantec as I recall.

"You can't reverse ransomware," I say. "You need the encryption key and it won't know that."

"This says it can!" This goes back and forth for a bit, and having finally given up to let him chase his tail, I pull out my phone and send an all staff email informing everyone of the situation, and to shut down their computers. No point in letting the ransomware run wildfire.

The CTO never wipes his stupid selling you something smile off his face, and announces to the room "It looks like you all have this under control. I have to leave to start my vacation for the long weekend. Let me know if you need anything!" And like that, he's gone. I'm stunned. I knew he wasn't good for anything but I'm still shocked that he would just shrug off a company wide ransomware attack.

The manager comes back in with a blank look on his face and says "It won't work, it needs to be installed before the ransomware attack." I say again "Because it needs to identify the key, which it gets in the process of it happening. Can we shut everything down now?"

Finally the manager starts managing enough to say "Shut it down." Our junior goes "Oh, it finished encrypting a while ago," and shuts the servers down. The manager goes to his desk - I find out later he's got his head down and is sobbing because he knows what this means.

In a roundtable I suggest at this point it's faster for us to just start wiping devices and sneakernetting MDT image reinstalls. "Do you know how long that'll take" our Sr Helpdesk asks. "Probably all weekend. But we have to get this back up, I'm up for it if you all are, but I'm not doing it alone. Get the MDT server restored first, and we can get running. Unplug from the network, reinstall from the image, and once that's done we can reconnect everything and get back online.

"Fine, let's do it."

The sysadmin finally began recovering servers, and by Monday everything was back up. We had a few gripes about users who "knew" their system wasn't infected and were mad that they lost all their data that they should've had on the file server, but for the most part it was about as smooth as it could be.

I found out later they only wanted ~$30,000 in Bitcoin. We ended up losing ~$350,000 a day because the CTO couldn't be bothered to engage seriously - or probably to even know what the actual impact would be. Of course the Manager, Sysadmin, and "Security" officer who wasn't really all took the fall for him to have his unbothered vacation.

And in a final very neat postscript, I found out much much later that it was the sysadmin that caused all this. He'd been using his regular account which was a FOREST ADMIN, including password, on random sketchy sites and he got us popped. They used our own SCCM against us which explained how it was able to go so wildfire.

16 Comments
2025/02/01
02:32 UTC

0

Is there a way to tell if your work is using a keylogger?

I'm in IT at my org but not sysadmin and I've never heard of this, but our union sent out a memo saying to always be careful because keyloggers can see everything you do at work. Maybe they just exaggerated and it's just web history, but it got me to thinking is there a way to tell? I don't see anything obvious in the processes and services, but then I would assume that a program like a keylogger perhaps tries to make itself inconspicuous.

42 Comments
2025/02/01
02:19 UTC

Back To Top