/r/linuxadmin

Photograph via snooOG

users voted

Expanding Linux SysAdmin knowledge

GUIDE to /r/linuxadmin:

  • Please consider that a new submission must help Linux SysAdmins
  • General blog/news/review posts belong in /r/linux
  • Articles/tutorials that simply reiterate what's in a manpage or a README, without adding significant value, are not useful
  • Inflammatory material doesn't help anyone but trolls

/r/linuxadmin aims to be a place where Linux SysAdmins can come together to get help and to support each other.

Related reddits:

Footnote:

Talk realtime on IRC at #/r/linuxadmin @ Freenode.

/r/linuxadmin

216,146 Subscribers

8

Increasing sda3 from sda

hey guys .how can i do this. I did know the way before but i forgot.

sda 8:0 0 3.5T 0 disk ├─sda1 8:1 0 1G 0 part /boot/efi ├─sda2 8:2 0 1G 0 part /boot └─sda3 8:3 0 1.2T 0 part

On sda3 im using LVM

Im using rhel 8.10

4 Comments
2024/12/01
10:34 UTC

1

What to expect in HPC/trading systems environments?

Hello, I'm considering a job change so I have been scouting for open Linux sysadmin opportunities in my corner of the world. Most of the traditional Linux roles I have seen so far are on 'high performance computing' and 'trading systems'.

What kinds of questions should I expect to receive during technical interviews with these kinds of roles? The job descriptions didn't reveal much difference to the usual 'sysadmin' role, aside from keywords such as 'high performance computing', 'trading systems', and a few familiar terms like Infiniband, network bonding, and some proprietary software for workload scheduling.

Thanks in advance.

14 Comments
2024/12/01
10:26 UTC

3

Accessing pfSense Web Configurator on Proxmox vs VMware Workstation: Networking Issue

I have a pfSense VM running on both VMware Workstation and Proxmox. Everything seems fine—on both setups, the WAN interface receives an IP from the local home router (using auto-bridge), and the LAN is configured. However, there's a difference in how I can access the pfSense web configurator:

  • In VMware Workstation, I can access the pfSense web configurator directly from the local host browser.
  • In Proxmox, I can only access the web configurator from a machine connected to the LAN network.

I can't figure out the difference in networking behavior between VMware Workstation and Proxmox that’s causing this. I would like to access the pfSense web configurator from the local PC (host machine) itself in the Proxmox setup, just like in VMware Workstation.

8 Comments
2024/11/29
21:28 UTC

0

Carve me a linux system administration roadmap.

I started with Linux OS. Then went to linux command line-->Bash scripting. Then learnt web servers (Apache HTTP/NGINX). I went to docker and kubernetes. And here's where I felt I was lacking and missing something. It has been 1 year and still I don't quite get docker and kubernetes. It leads me to the conclusion that I am missing some preriquisites.

I am completely off track of everything else from docker and kubernetes.

Thus, I want to know what's that? Is that yamls? Is that ansible?

15 Comments
2024/11/29
12:19 UTC

10

Transparent TLS and HTTP proxy that serves on all 65535 ports

Goshkan, a transparent TLS and HTTP proxy that operates on all 65535 ports. with domain regex whitelisting, payload inspection, low memory usage, and a REST API for managing domain filters.

  • TLS & HTTP on the same port: Supports payload inspection and connection management.
  • Low memory footprint: Handles traffic efficiently with minimal memory usage.
  • Regex domain filtering: Filters traffic based on domain regex patterns.
  • REST API: Allows adding/removing domains programmatically.
  • Operating on all ports: Uses iptables for redirection across all ports.
  • DNAT friendly: Can detect the actual destination port from the conntrack table.
  • Written in Go: Uses Golang standard packages, with the exception of the MySQL driver.

https://github.com/Sina-Ghaderi/goshkan

10 Comments
2024/11/28
14:35 UTC

15

Monitoring solution for two linux servers

Hey,

I'm looking for a monitoring solution for two ubuntu servers. Seems to me there is a lot of different solution and I'm getting a bit lost. I'm looking to monitor things such as basic hardware usage, users logs and commands, open ports, security...

We use Entra ID a lot. I wonder if it's worth monitoring those servers with Azure Arc & Azure Monitor for simplicity sakes. Seems rather cheap for two servers. We also already use Defender for all our endpoints (except those servers).

What do you guys use for monitoring ? Can Azure and Defender works well with Linux servers ?

22 Comments
2024/11/28
06:49 UTC

8

How do you automate environment set up pre-provisioning?

Forgive the ignorance, please correct anything that is wrong or fill in any gaps I'm missing.

As I understand it, you use a configuration management system like Ansible, Chef, or Puppet for the more day to day management of your systems; updating software, firewall rules, etc. Before we can think about that though, we have mention provisioning tools like Terraform or OpenTofu, who initialize the virtual systems that get managed by your config management system. My main query comes in as 'what happens before that point?' I recognize that a lot of the time that responsibility is schlepped off to the cloud providers and your provisioning tool just interacts with them, but what about those companies that have on-prem resources? How are those baremetal systems bootstrapped? I imagine those companies aren't manually installing OSs prior to using a provisioning tool? The only thing I can think of would be something like booting the baremetal servers from a pxe server containing a customized image. Am I off base?

40 Comments
2024/11/28
05:42 UTC

7

IP Forwarding two subnets/networks together - Can NetworkManager do this or have I been trying to use a fork to drink soup?

11 Comments
2024/11/27
22:59 UTC

6

How two processes communicate via an SSH stream?

Hi,

how two processes communicate via SSH stream?

Well, I'm speaking of rsync via SSH. With this simple command:

rsync -avz user@address:/home/user ./backup

rsync create an ssh session and on the other side "rsync --server ...." is executed that wait for protocol command. But How that works really? How the 2 processes can communicate between them via SSH?

To understand this I created a simple python script that try to read data sent from the other side of the connection, simply reading stdin and if it found "test" command it should print a string. Here the code:

import sys

for line in sys.stdin:

if(line[:-1] == "exit"):

exit(0)

elif(line[:-1] == "test"):

print("test received")

Running 'ssh user@address "pythonscript.py"' it does not work, no output from the script because it seems not able to read from the ssh connection, maybe the script should not read from stdin but from another "source"? I don't know..

I tried using ssh -t that create a pseudo terminal and with this method I can send command/data to my script.

Another way I found is SSH Tunnel (port forwarding) to permit two program to talk via network sockets.

But I can't understand how rsync can communicate with the server part via SSH. There is something that is piped or other? I tried with strace but this is a huge output of what rsync does and what ssh does.

Any tips/help/suggestion will be appreciated.

Thank you in advance.

16 Comments
2024/11/27
11:15 UTC

13

Best Udemy course for RHCSA EX200

I am looking for Udemy course which is best for RHCSA EX200.

Please let me know if any course or material I need to refer for this exam.

4 Comments
2024/11/27
08:12 UTC

9

Rsync backup with hardlink (--link-dest): the hardlink farm problem

Hi,

I'm using rsync + python to perform backups using hardlink (--link-dest option of rsync). I mean: I run the first full backup and other backups with --link-dest option. It work very well, it does not create hardlink of the original copy but hardlink on the first backup and so on.

I'm dealing with a statement "using rsync with hardlink, you will have an hardlink farm".

What are drawbacks of having an "hardlink farm"?

Thank you in advance.

35 Comments
2024/11/26
18:16 UTC

7

Socket pairs getting desynced, TIME_WAIT issues, SYN cookies..

Hi. I wasn't sure which subreddit would be most appropriate, and where there might be enough users to get some insight, but I'll try my luck here!

For quick context: I'm a developer, with broad rather than deep experience. I've maintained and developed infrastructure, both cloud and non-cloud ones. Mainly with Linux servers.

So, the issue: One client noticed they had to restart our product every few days, as it ran out of file handles.

In subsequent load tests, we noticed that under some traffic patterns, some sockets and their associated connection are left on one side in TIME_WAIT state, while on the other side, the connection is in ESTABLISHED. While in ESTABLISHED, it sends a keepalive ACK packet and the TIME_WAIT MLS timer resets.

I was a little bit surprised to find that the timer for TIME_WAIT will reset on traffic. It seems like this is hard-coded behavior in the Linux kernel, and can not be modified.

On further testing, it seems that the issue is SYN cookies being on, and here the issue seems to have been the same: https://medium.com/appsflyerengineering/the-story-of-the-tcp-connections-that-refused-to-die-ee1726615d29

We can fix this for now by disabling SYN cookies and/or by tuning the keepalive values, but this led me to another realization: Couldn't a misbehaving client - whether due to a bug or deliberately as a form of DoS attack - attempt to deliberately create a similar situation?

I'd suppose that the question thus is, are there some fairly standard ways of e.g. cleaning up sockets in active close state if file handles are close to being exhausted? What kind of strategies are common for dealing with these sort of situations?

2 Comments
2024/11/26
10:28 UTC

0

Can't ping github.com

Hi all, I recently installed ubuntu server 24.04.1 LTS on an old computer, and can't seem to connect to github at all. I can't use ssh or https. DNS seems to be working fine, because the IP address that it finds works when I use other computers to ping it.

I'm using Network Manager as that was the only way I could get my old wifi card to work.

Here's a screenshot of my firewall status:

https://preview.redd.it/fchz0rqpv73e1.png?width=615&format=png&auto=webp&s=d8d1541b8824c737607edca48fb7b87d434f6831

Thanks in advance for any help.

7 Comments
2024/11/26
09:44 UTC

8

Is it possible to clone an OS disk to smaller disk?

Hi,

Just wanted to ask, i have 100GB OS disk with wifh xfs filesytem, here is the setup

NAME                   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda                      8:0    0  100G  0 disk
├─sda1                   8:1    0  500M  0 part /boot/efi
├─sda2                   8:2    0    1G  0 part /boot
└─sda3                   8:3    0 98.5G  0 part
  ├─Vol00-LVSlash 253:0    0   20G  0 lvm  /
  ├─Vol00-LVHome  253:2    0   10G  0 lvm  /home
  ├─Vol00-LVLog   253:3    0   10G  0 lvm  /var/log
  └─Vol00-LVVar   253:4    0   10G  0 lvm  /var

/dev/sda3 has still 48.5 GB free space. all filesystems use less than 25% space.

Is it possible to clone this to a 50GB or 60GB disk? if not what are my options?

21 Comments
2024/11/25
23:42 UTC

2

Missing dev of network card

Hi, excuse me if this is a noob question but I never had to deal with something like this.

My server (Debian 12) has two network cards and as we are having issues with one of them and a PVE kernel upgrade, we need to test through the other one. Our second Realtek card does not list an interface name. I have a enp6s0 but nothing on the other. I can configure networks, but never had to face not having a hardware interface name for one. Unsure if this might be a hardware, bios problem or some missing configuration.

#lspci | grep "Ethernet"
06:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03)
07:00.1 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 1a)

#hwinfo --short --netcard
network:
  enp6s0               Intel I211 Gigabit Network Connection
                       Realtek RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller

# lshw -C network -short
H/W path                Device          Class          Description
==================================================================
/0/100/1.2/0/3/0        enp6s0          network        I211 Gigabit Network Connection
/0/100/1.2/0/4/0.1                      network        RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller

What can I do to start the interface and that it gets an interface name assigned?

Thanks.

(Edited to clarify the question)

4 Comments
2024/11/25
20:20 UTC

1

Gocryptfs vs CryFS

Hi,

I use gocryptfs to encrypt my backups and found cryfs that seems a good software. I tried it a bit but not so much to have a good comparison. It seems fast like gocryptfs, it does not report file size because it saves on "blocks", it creates much more file vs gocryptfs that are update when more data reach encrypted directory so in case of sync on cloud service I could resync a very big chunk of data for a single file modification..other things don't come to my mind.

Do you use cryfs and in what way it is better vs gocryptfs?

Thank you in advance

6 Comments
2024/11/25
18:45 UTC

10

Rhel 9 desktop screen idle doesn't register terminal as activity

I set the screen to auto-lock (employer workstation -- required) in the settings, but most of my work is still terminal, and the screen lock seems to ignore me just typing and running things in terminal. I have to jiggle the mouse every so often or the screen blanks and locks.

I'm using the default gnome/Wayland for workstation. Is there a setting buried somewhere in /etc that the screen lock uses to determine what inputs constitute "activity"?

2 Comments
2024/11/25
17:45 UTC

28

Setup Centralized Logging with Rsyslog in 6 minutes

2 Comments
2024/11/24
23:45 UTC

0

Powerful Command line tools for DevOps: Nushell and Jc

Revolutionize Your DevOps Workflow! 💥

Tired of drowning in unstructured text data? 🌊 Introducing Nushell and Jc, two game-changing tools that will transform the way you work with data! 🔥

Nushell: The Modern Marvel 🤖 Rewrites command-line tools to export structured data. 💡 Say goodbye to tedious text processing!

Jc: The JSON Converter 📈 Converts legacy Linux command output into JSON format. Simplify complex tasks and collaborate more effectively! 🤝

Benefits Are Endless! 🌈

Gain efficiency, simplify scripting, improve collaboration, and reduce errors with Nushell and Jc.

Read the Full Article Here: https://cloudnativeengineer.substack.com/p/powerful-command-line-tools-for-devops 📄

20 Comments
2024/11/24
21:17 UTC

1

T14s gen 2 laggy wifi

0 Comments
2024/11/24
12:13 UTC

11

Want to learn to make a cicd project using jenkins, gitlab, harbour and k3s. Any tutorials?

I've been working in tech as a support engineer since 2 years (About to be) and today I feel like doing a project in cicd.

In my current company, cicd isn't implemented but it's done manually.(I feel like that I am not sure lol)

I know code is put in gitlab. Then it's built in jenkins. Then it's put to harbour image repository. Then it's deployed on kubernetes. (That's all I know as a support engineer as the devops team does everything.)

I want someone to guide and make a complete end to end project on ci cd. I'd be grateful if you can recommend some paid courses from any platform. As learning by projects is the best way to learn.

Edit: I just installed jenkins in my linux server. Now what I want is write some small code in and host in self hosted gitlab server (in same linux server)...Then do CI with jenkins

6 Comments
2024/11/24
08:14 UTC

3

Load ipset on reboot, before iptables - Ubuntu?

Do you have any best practices, examples of loading ipset rules on boot on Ubuntu?

Must be before iptables, otherwise iptables(-restore) will fail.

4 Comments
2024/11/24
08:02 UTC

5

Help route internet from usb tether to lan - nat, routes & nftables

Im trying to setup my box to route internet from end0 (192.168.1.6) to internet on usb0 (dhcp). Im running dns & dhcp via docker adguard - but assume thats not working for now because once the nftable rules are applied I cannot access their web interfaces. But for now ping with ip is okay.

With my current setup I can ping the internet from the ‘router’ via the interface usb0. But I cannot ping from the interface end0.

ping 8.8.8.8 -I usb0 ← works
ping 8.8.8.8 -I end0 ← Destination Host Unreachable

Do I need to setup any static routes? Or should nftables handle all the routing? Ive tried several guides with various nftable rules, but none of them work:

my network config:

usb0:
[Match]
Name=usb0

[Network]
DHCP=yes


end0:
[Match]
Name=end0

[Network]
Address=192.168.1.6/24

my nftables:

table inet filter {
        chain input {
                type filter hook input priority filter; policy accept;
        }

        chain forward {
                type filter hook forward priority filter; policy accept;
                iif "end0" oif "usb0" accept
                iif "usb0" oif "end0" accept
        }

        chain output {
                type filter hook output priority filter; policy accept;
        }
}
table ip nat {
        chain prerouting {
                type nat hook prerouting priority filter; policy accept;
        }

        chain postrouting {
                type nat hook postrouting priority srcnat; policy accept;
                oif "usb0" masquerade
        }
}

my routes:

default via 192.168.102.208 dev usb0 
default via 192.168.102.208 dev usb0 proto dhcp src 192.168.102.114 metric 1024 
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown <- docker?
172.18.0.0/16 dev br-cc00a7d88795 proto kernel scope link src 172.18.0.1 <- docker?
192.168.1.0/24 dev end0 proto kernel scope link src 192.168.1.6 
192.168.102.0/24 dev usb0 proto kernel scope link src 192.168.102.114 metric 1024 
192.168.102.208 dev usb0 proto dhcp scope link src 192.168.102.114 metric 1024 
5 Comments
2024/11/23
09:00 UTC

23

Best way to limit total memory used by all users on a shared multi-user system

Our site has many CentOS7, Rocky8/9 linux systems that are shared by many users concurrently via ssh login for random interactive uses. Many of these are large 128GB+ desktops at one person in a a groups desk where that person logins in person but many other users in the group SSH in to that desktop to run various analysis programs and development.

Anyway, one thing that happens a lot is one user will run a MATLAB or other program that consuses all the RAM in the box slowing it down to a crawl for all others. Eventually the kernel implements its OOM procedure. However, many system processes, though not killed by the OOM procedure get in a stuck non-operating state.

One of these is SSSD the main account services daemon which does not recover and then prevents any new logins and hangs other processes on things like user name/id lookups. One can restart sssd to fix it but one cannot ssh to the box or even login locally to do this. So most of the time we have to hard powercycle the box.

One attempt I made at "fixing" this was to create the following rsyslog configuration in /etc/rsyslog.d/oom-sssd-restart.conf

:msg, contains, "was terminated by own WATCHDOG" ^/usr/etc/sssd-restart.sh

as one usually sees that message in /var/log/messages when sssd gets in its hung state but this has only worked about 50% of the time

Ultimately, I want to make sure that 4GB or so of the RAM of each system is reserved only for system processes (UID < 1000) or just limit RAM to 96% of the systems ram to users with UID > 1000. Is there any simple and accepted way to do this? I am NOT looking for a per user memory limit via the /etc/security/limits.d/ system. That does not work for what I want.

One thing I am looking at is using cgroup slices and running

systemctl set-property user.slice MemoryHigh=120G

for example on a 128G system. It is unclear to me if this requires cgroups v2 meaning changing GRUB on all boxes to have kernel paramater systemd.unified_cgroup_hierarchy=1 and rebooting them.

BTW, I do use SLURM on a HPC cluster and consider that a too heavy handed and difficult solution for an interactive user desktop shared by users where local GUI login is used.

19 Comments
2024/11/22
14:02 UTC

3

Question about backup encryption

Hi,

suppose you have a server in your company that backups several server (remote and local) and data on server are not encrypted. The backup can use whatever backup solution (bacula, bareos, veeam, acronis, borgbackup, restic, kopia, rsync...) and that it encrypt backups. Being an automatic operation the encryption key(s) is stored on the backup server and used when the backup start. In this way if an attacker take control of backup server he can stole the key, data and decrypt them or worst corrupt data without need of decrypt them.

It can be usefull if you use tape and store them, or when disks are full and they are swapped and stored.

I can understand when you need to save them offsite (like on S3 or another solution) and encryption is a must, but as said, is it worth encrypt local backups considering the previous scenario?

In what case having encrypted backup is usefull?

Thank you in advance.

11 Comments
2024/11/22
08:54 UTC

5

RHEL 8 - NM doesn't see Wi-Fi card

I need some help getting Wi-Fi working on a Linux machine. It's a pretty simple machine that has two Ethernet interfaces and a single Wi-Fi NIC. Currently we are using the network-scripts ifcfg files for the static IP addressing on the two Ethernet NICs and DHCP for Wi-Fi. All the networking functions work as designed, but network manager cannot see the Wi-Fi interface (wls2). I've been able to get the Wi-Fi card to pull an IP from the DHCP server utilizing wpa-supplicant and DHCP client. The problem I keep encountering is that the route table is not getting updated for the Wi-Fi connection. From what I read online, NetworkManager is in charge of managing routes and here in lies the problem. Nmcli shows the Wi-Fi interface as "unavailable" and nothing I do seems to bring it online for network manager. Both of the Ethernet interfaces are working fine with nmcli. I'm really at a loss on how to resolve the problem.

OS: RHEL 8.10

Thanks for the help!

0 Comments
2024/11/21
22:36 UTC

6

Resources for teaching "Engineer Mindset"?

I have a new starter at work, and I need to try and fast track them as much as I can from a 1st/2nd line background to more of an Engineer mindset. Things like:

  • Critical Thinking Processes
  • Independent Investigative Troubleshooting
  • Root Cause Analysis
  • General Thoroughness

I appreciate to a degree some of this only comes from hard earned experience, but can anybody suggest me any online resources that might be helpful to give them to help them adopt/progress to a more 3rd line mindset?

Thanks in advance.

EDIT: Possibly "fast track" was a poor choice of words here. I'm not looking to alter anybody's brain chemistry in a day, I'm just looking for some teaching resources on the softer skills involved in being a Linux Engineer.

11 Comments
2024/11/19
10:56 UTC

5

Does BTRFS allow hot snapshots?

Hi!

I'm going to install a new server and I wanted to know if using BTRFS I'll have the possibility of making hot snapshots.

I usually use debian with ext4 and docker but I would like to be able to take snapshots of the entire system

17 Comments
2024/11/19
09:54 UTC

49

A day in the life of a linuxadmin

Hey, was thinking if you want to share a day in the life of your current job.

What do you do? How long hours do you work? Do you get called in weekends and evenings? What’s your title? Small or large company? Pros/cons? How would you like it instead? Maybe this can be your guideline

It would be interesting to see different aspects of the Linuxadmins.

There are some older threads here already but times have changed and lots of new people here as well.

20 Comments
2024/11/19
06:43 UTC

Back To Top