/r/linuxadmin
users voted
GUIDE to /r/linuxadmin:
/r/linuxadmin aims to be a place where Linux SysAdmins can come together to get help and to support each other.
Related reddits:
Footnote:
Talk realtime on IRC at #/r/linuxadmin @ Freenode.
/r/linuxadmin
hey guys .how can i do this. I did know the way before but i forgot.
sda 8:0 0 3.5T 0 disk ├─sda1 8:1 0 1G 0 part /boot/efi ├─sda2 8:2 0 1G 0 part /boot └─sda3 8:3 0 1.2T 0 part
On sda3 im using LVM
Im using rhel 8.10
Hello, I'm considering a job change so I have been scouting for open Linux sysadmin opportunities in my corner of the world. Most of the traditional Linux roles I have seen so far are on 'high performance computing' and 'trading systems'.
What kinds of questions should I expect to receive during technical interviews with these kinds of roles? The job descriptions didn't reveal much difference to the usual 'sysadmin' role, aside from keywords such as 'high performance computing', 'trading systems', and a few familiar terms like Infiniband, network bonding, and some proprietary software for workload scheduling.
Thanks in advance.
I have a pfSense VM running on both VMware Workstation and Proxmox. Everything seems fine—on both setups, the WAN interface receives an IP from the local home router (using auto-bridge), and the LAN is configured. However, there's a difference in how I can access the pfSense web configurator:
I can't figure out the difference in networking behavior between VMware Workstation and Proxmox that’s causing this. I would like to access the pfSense web configurator from the local PC (host machine) itself in the Proxmox setup, just like in VMware Workstation.
I started with Linux OS. Then went to linux command line-->Bash scripting. Then learnt web servers (Apache HTTP/NGINX). I went to docker and kubernetes. And here's where I felt I was lacking and missing something. It has been 1 year and still I don't quite get docker and kubernetes. It leads me to the conclusion that I am missing some preriquisites.
I am completely off track of everything else from docker and kubernetes.
Thus, I want to know what's that? Is that yamls? Is that ansible?
Goshkan, a transparent TLS and HTTP proxy that operates on all 65535 ports. with domain regex whitelisting, payload inspection, low memory usage, and a REST API for managing domain filters.
Hey,
I'm looking for a monitoring solution for two ubuntu servers. Seems to me there is a lot of different solution and I'm getting a bit lost. I'm looking to monitor things such as basic hardware usage, users logs and commands, open ports, security...
We use Entra ID a lot. I wonder if it's worth monitoring those servers with Azure Arc & Azure Monitor for simplicity sakes. Seems rather cheap for two servers. We also already use Defender for all our endpoints (except those servers).
What do you guys use for monitoring ? Can Azure and Defender works well with Linux servers ?
Forgive the ignorance, please correct anything that is wrong or fill in any gaps I'm missing.
As I understand it, you use a configuration management system like Ansible, Chef, or Puppet for the more day to day management of your systems; updating software, firewall rules, etc. Before we can think about that though, we have mention provisioning tools like Terraform or OpenTofu, who initialize the virtual systems that get managed by your config management system. My main query comes in as 'what happens before that point?' I recognize that a lot of the time that responsibility is schlepped off to the cloud providers and your provisioning tool just interacts with them, but what about those companies that have on-prem resources? How are those baremetal systems bootstrapped? I imagine those companies aren't manually installing OSs prior to using a provisioning tool? The only thing I can think of would be something like booting the baremetal servers from a pxe server containing a customized image. Am I off base?
Hi,
how two processes communicate via SSH stream?
Well, I'm speaking of rsync via SSH. With this simple command:
rsync -avz user@address:/home/user ./backup
rsync create an ssh session and on the other side "rsync --server ...." is executed that wait for protocol command. But How that works really? How the 2 processes can communicate between them via SSH?
To understand this I created a simple python script that try to read data sent from the other side of the connection, simply reading stdin and if it found "test" command it should print a string. Here the code:
import sys
for line in sys.stdin:
if(line[:-1] == "exit"):
exit(0)
elif(line[:-1] == "test"):
print("test received")
Running 'ssh user@address "pythonscript.py"' it does not work, no output from the script because it seems not able to read from the ssh connection, maybe the script should not read from stdin but from another "source"? I don't know..
I tried using ssh -t that create a pseudo terminal and with this method I can send command/data to my script.
Another way I found is SSH Tunnel (port forwarding) to permit two program to talk via network sockets.
But I can't understand how rsync can communicate with the server part via SSH. There is something that is piped or other? I tried with strace but this is a huge output of what rsync does and what ssh does.
Any tips/help/suggestion will be appreciated.
Thank you in advance.
I am looking for Udemy course which is best for RHCSA EX200.
Please let me know if any course or material I need to refer for this exam.
Hi,
I'm using rsync + python to perform backups using hardlink (--link-dest option of rsync). I mean: I run the first full backup and other backups with --link-dest option. It work very well, it does not create hardlink of the original copy but hardlink on the first backup and so on.
I'm dealing with a statement "using rsync with hardlink, you will have an hardlink farm".
What are drawbacks of having an "hardlink farm"?
Thank you in advance.
Hi. I wasn't sure which subreddit would be most appropriate, and where there might be enough users to get some insight, but I'll try my luck here!
For quick context: I'm a developer, with broad rather than deep experience. I've maintained and developed infrastructure, both cloud and non-cloud ones. Mainly with Linux servers.
So, the issue: One client noticed they had to restart our product every few days, as it ran out of file handles.
In subsequent load tests, we noticed that under some traffic patterns, some sockets and their associated connection are left on one side in TIME_WAIT state, while on the other side, the connection is in ESTABLISHED. While in ESTABLISHED, it sends a keepalive ACK packet and the TIME_WAIT MLS timer resets.
I was a little bit surprised to find that the timer for TIME_WAIT will reset on traffic. It seems like this is hard-coded behavior in the Linux kernel, and can not be modified.
On further testing, it seems that the issue is SYN cookies being on, and here the issue seems to have been the same: https://medium.com/appsflyerengineering/the-story-of-the-tcp-connections-that-refused-to-die-ee1726615d29
We can fix this for now by disabling SYN cookies and/or by tuning the keepalive values, but this led me to another realization: Couldn't a misbehaving client - whether due to a bug or deliberately as a form of DoS attack - attempt to deliberately create a similar situation?
I'd suppose that the question thus is, are there some fairly standard ways of e.g. cleaning up sockets in active close state if file handles are close to being exhausted? What kind of strategies are common for dealing with these sort of situations?
Hi all, I recently installed ubuntu server 24.04.1 LTS on an old computer, and can't seem to connect to github at all. I can't use ssh or https. DNS seems to be working fine, because the IP address that it finds works when I use other computers to ping it.
I'm using Network Manager as that was the only way I could get my old wifi card to work.
Here's a screenshot of my firewall status:
Thanks in advance for any help.
Hi,
Just wanted to ask, i have 100GB OS disk with wifh xfs filesytem, here is the setup
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 100G 0 disk
├─sda1 8:1 0 500M 0 part /boot/efi
├─sda2 8:2 0 1G 0 part /boot
└─sda3 8:3 0 98.5G 0 part
├─Vol00-LVSlash 253:0 0 20G 0 lvm /
├─Vol00-LVHome 253:2 0 10G 0 lvm /home
├─Vol00-LVLog 253:3 0 10G 0 lvm /var/log
└─Vol00-LVVar 253:4 0 10G 0 lvm /var
/dev/sda3 has still 48.5 GB free space. all filesystems use less than 25% space.
Is it possible to clone this to a 50GB or 60GB disk? if not what are my options?
Hi, excuse me if this is a noob question but I never had to deal with something like this.
My server (Debian 12) has two network cards and as we are having issues with one of them and a PVE kernel upgrade, we need to test through the other one. Our second Realtek card does not list an interface name. I have a enp6s0 but nothing on the other. I can configure networks, but never had to face not having a hardware interface name for one. Unsure if this might be a hardware, bios problem or some missing configuration.
#lspci | grep "Ethernet"
06:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03)
07:00.1 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 1a)
#hwinfo --short --netcard
network:
enp6s0 Intel I211 Gigabit Network Connection
Realtek RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller
# lshw -C network -short
H/W path Device Class Description
==================================================================
/0/100/1.2/0/3/0 enp6s0 network I211 Gigabit Network Connection
/0/100/1.2/0/4/0.1 network RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller
What can I do to start the interface and that it gets an interface name assigned?
Thanks.
(Edited to clarify the question)
Hi,
I use gocryptfs to encrypt my backups and found cryfs that seems a good software. I tried it a bit but not so much to have a good comparison. It seems fast like gocryptfs, it does not report file size because it saves on "blocks", it creates much more file vs gocryptfs that are update when more data reach encrypted directory so in case of sync on cloud service I could resync a very big chunk of data for a single file modification..other things don't come to my mind.
Do you use cryfs and in what way it is better vs gocryptfs?
Thank you in advance
I set the screen to auto-lock (employer workstation -- required) in the settings, but most of my work is still terminal, and the screen lock seems to ignore me just typing and running things in terminal. I have to jiggle the mouse every so often or the screen blanks and locks.
I'm using the default gnome/Wayland for workstation. Is there a setting buried somewhere in /etc that the screen lock uses to determine what inputs constitute "activity"?
Revolutionize Your DevOps Workflow! 💥
Tired of drowning in unstructured text data? 🌊 Introducing Nushell and Jc, two game-changing tools that will transform the way you work with data! 🔥
Nushell: The Modern Marvel 🤖 Rewrites command-line tools to export structured data. 💡 Say goodbye to tedious text processing!
Jc: The JSON Converter 📈 Converts legacy Linux command output into JSON format. Simplify complex tasks and collaborate more effectively! 🤝
Benefits Are Endless! 🌈
Gain efficiency, simplify scripting, improve collaboration, and reduce errors with Nushell and Jc.
Read the Full Article Here: https://cloudnativeengineer.substack.com/p/powerful-command-line-tools-for-devops 📄
I've been working in tech as a support engineer since 2 years (About to be) and today I feel like doing a project in cicd.
In my current company, cicd isn't implemented but it's done manually.(I feel like that I am not sure lol)
I know code is put in gitlab. Then it's built in jenkins. Then it's put to harbour image repository. Then it's deployed on kubernetes. (That's all I know as a support engineer as the devops team does everything.)
I want someone to guide and make a complete end to end project on ci cd. I'd be grateful if you can recommend some paid courses from any platform. As learning by projects is the best way to learn.
Edit: I just installed jenkins in my linux server. Now what I want is write some small code in and host in self hosted gitlab server (in same linux server)...Then do CI with jenkins
Do you have any best practices, examples of loading ipset rules on boot on Ubuntu?
Must be before iptables, otherwise iptables(-restore) will fail.
Im trying to setup my box to route internet from end0 (192.168.1.6) to internet on usb0 (dhcp). Im running dns & dhcp via docker adguard - but assume thats not working for now because once the nftable rules are applied I cannot access their web interfaces. But for now ping with ip is okay.
With my current setup I can ping the internet from the ‘router’ via the interface usb0. But I cannot ping from the interface end0.
ping 8.8.8.8 -I usb0 ← works
ping 8.8.8.8 -I end0 ← Destination Host Unreachable
Do I need to setup any static routes? Or should nftables handle all the routing? Ive tried several guides with various nftable rules, but none of them work:
my network config:
usb0:
[Match]
Name=usb0
[Network]
DHCP=yes
end0:
[Match]
Name=end0
[Network]
Address=192.168.1.6/24
my nftables:
table inet filter {
chain input {
type filter hook input priority filter; policy accept;
}
chain forward {
type filter hook forward priority filter; policy accept;
iif "end0" oif "usb0" accept
iif "usb0" oif "end0" accept
}
chain output {
type filter hook output priority filter; policy accept;
}
}
table ip nat {
chain prerouting {
type nat hook prerouting priority filter; policy accept;
}
chain postrouting {
type nat hook postrouting priority srcnat; policy accept;
oif "usb0" masquerade
}
}
my routes:
default via 192.168.102.208 dev usb0
default via 192.168.102.208 dev usb0 proto dhcp src 192.168.102.114 metric 1024
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown <- docker?
172.18.0.0/16 dev br-cc00a7d88795 proto kernel scope link src 172.18.0.1 <- docker?
192.168.1.0/24 dev end0 proto kernel scope link src 192.168.1.6
192.168.102.0/24 dev usb0 proto kernel scope link src 192.168.102.114 metric 1024
192.168.102.208 dev usb0 proto dhcp scope link src 192.168.102.114 metric 1024
Our site has many CentOS7, Rocky8/9 linux systems that are shared by many users concurrently via ssh login for random interactive uses. Many of these are large 128GB+ desktops at one person in a a groups desk where that person logins in person but many other users in the group SSH in to that desktop to run various analysis programs and development.
Anyway, one thing that happens a lot is one user will run a MATLAB or other program that consuses all the RAM in the box slowing it down to a crawl for all others. Eventually the kernel implements its OOM procedure. However, many system processes, though not killed by the OOM procedure get in a stuck non-operating state.
One of these is SSSD the main account services daemon which does not recover and then prevents any new logins and hangs other processes on things like user name/id lookups. One can restart sssd to fix it but one cannot ssh to the box or even login locally to do this. So most of the time we have to hard powercycle the box.
One attempt I made at "fixing" this was to create the following rsyslog configuration in /etc/rsyslog.d/oom-sssd-restart.conf
:msg, contains, "was terminated by own WATCHDOG" ^/usr/etc/sssd-restart.sh
as one usually sees that message in /var/log/messages when sssd gets in its hung state but this has only worked about 50% of the time
Ultimately, I want to make sure that 4GB or so of the RAM of each system is reserved only for system processes (UID < 1000) or just limit RAM to 96% of the systems ram to users with UID > 1000. Is there any simple and accepted way to do this? I am NOT looking for a per user memory limit via the /etc/security/limits.d/ system. That does not work for what I want.
One thing I am looking at is using cgroup slices and running
systemctl set-property user.slice MemoryHigh=120G
for example on a 128G system. It is unclear to me if this requires cgroups v2 meaning changing GRUB on all boxes to have kernel paramater systemd.unified_cgroup_hierarchy=1 and rebooting them.
BTW, I do use SLURM on a HPC cluster and consider that a too heavy handed and difficult solution for an interactive user desktop shared by users where local GUI login is used.
Hi,
suppose you have a server in your company that backups several server (remote and local) and data on server are not encrypted. The backup can use whatever backup solution (bacula, bareos, veeam, acronis, borgbackup, restic, kopia, rsync...) and that it encrypt backups. Being an automatic operation the encryption key(s) is stored on the backup server and used when the backup start. In this way if an attacker take control of backup server he can stole the key, data and decrypt them or worst corrupt data without need of decrypt them.
It can be usefull if you use tape and store them, or when disks are full and they are swapped and stored.
I can understand when you need to save them offsite (like on S3 or another solution) and encryption is a must, but as said, is it worth encrypt local backups considering the previous scenario?
In what case having encrypted backup is usefull?
Thank you in advance.
I need some help getting Wi-Fi working on a Linux machine. It's a pretty simple machine that has two Ethernet interfaces and a single Wi-Fi NIC. Currently we are using the network-scripts ifcfg files for the static IP addressing on the two Ethernet NICs and DHCP for Wi-Fi. All the networking functions work as designed, but network manager cannot see the Wi-Fi interface (wls2). I've been able to get the Wi-Fi card to pull an IP from the DHCP server utilizing wpa-supplicant and DHCP client. The problem I keep encountering is that the route table is not getting updated for the Wi-Fi connection. From what I read online, NetworkManager is in charge of managing routes and here in lies the problem. Nmcli shows the Wi-Fi interface as "unavailable" and nothing I do seems to bring it online for network manager. Both of the Ethernet interfaces are working fine with nmcli. I'm really at a loss on how to resolve the problem.
OS: RHEL 8.10
Thanks for the help!
I have a new starter at work, and I need to try and fast track them as much as I can from a 1st/2nd line background to more of an Engineer mindset. Things like:
I appreciate to a degree some of this only comes from hard earned experience, but can anybody suggest me any online resources that might be helpful to give them to help them adopt/progress to a more 3rd line mindset?
Thanks in advance.
EDIT: Possibly "fast track" was a poor choice of words here. I'm not looking to alter anybody's brain chemistry in a day, I'm just looking for some teaching resources on the softer skills involved in being a Linux Engineer.
Hi!
I'm going to install a new server and I wanted to know if using BTRFS I'll have the possibility of making hot snapshots.
I usually use debian with ext4 and docker but I would like to be able to take snapshots of the entire system
Hey, was thinking if you want to share a day in the life of your current job.
What do you do? How long hours do you work? Do you get called in weekends and evenings? What’s your title? Small or large company? Pros/cons? How would you like it instead? Maybe this can be your guideline
It would be interesting to see different aspects of the Linuxadmins.
There are some older threads here already but times have changed and lots of new people here as well.