After more than twenty years on UNIX systems, it seems like the right time to begin a guide to Linux systems administration, and what better place than on MangoLassi where we can make it into a living, growing document.

An Introduction: Why Linux System Administration

• Linux: Collected Educational Resources
• Why Desktops Are Not a Good Learning Tool
• Avoid the Raspberry Pi for Learning Linux Server Administration
• Linux: What is a Distro
• Choosing a Linux Distribution for Business
• What Is UNIX, and How Does Linux Fit In
• What Makes Something a Linux OS
• Why Does Open Source Matter
• Understanding Linux Release Schedules
• Understanding the Relationships of Red Hat's RHEL, CentOS and Fedora Distributions
• Understanding the Suse Distributions Suse Linux, OpenSuse Leap and OpenSuse Tumbleweed
• Understanding the Debian and Ubuntu Linux Distributions
• Linux Architectural Variety
• Linux and Virtualization
• From Windows to UNIX: Monolithic to Modular Design

---

Basics of Linux Administration

Getting Our Hands Dirty: Learning Our Way Around

• Installing Our First Linux Operating System for Learning Systems Administration
• Linux: The Lay of the Land, Filesystem Hierarchy Standard
• Accessing a Linux Server using SSH
• Linux: Working with Files
• From Windows to UNIX: File Extensions
• From Windows to UNIX: Case Sensitivity
• Linux: File Colors
• Linux: Standard Command Structures
• Linux: Aliases
• Linux: Basic Working with Text Files
• Linux: Finding Files
• From Windows to UNIX: Text File Formats
• Linux: Symbolically Linking Files
• UNIX: Users and Groups
  • UNIX: The root user
  • UNIX: The /etc/passwd File in Depth
  • UNIX: The /etc/group File in Depth
  • UNIX: The /etc/shadow File in Depth
  • Linux: Tools for Managing Local Users
  • Linux: Tools for Managing Local Groups
  • UNIX: Switching Users with su
  • UNIX: sudo
  • UNIX: The sudoers file
• UNIX: Basic File Permissions
  • UNIX: File Permission Octets
• UNIX: Extended File Permissions and Attributes
• UNIX: Everything is a File (Descriptor)
• Linux: Shells
• System Administration: Standard Tools
• UNIX: Path
• Linux: Home Directories
• Linux: Common Filesystems
  • From Windows to Linux: Filesystem Culture
• Linux: Text Editing
• UNIX: X Windows and the Graphical Desktop

Packages and Installation

• From Windows to Linux: Installation Culture
• Linux: Software Installation
  Red Hat and the RPM World
  * Linux: RPM Package Management
  • Linux: Querying with RPM
  • Linux: Installing with RPM
  • Linux: Uninstalling with RPM

* [Linux: YUM Package Management](http://mangolassi.it/topic/8629)
  * Linux: YUM Repositories
  * [Linux: Installing with YUM](http://mangolassi.it/topic/8799/)
  * [Linux: Updating with YUM](http://mangolassi.it/topic/8800/)
  * [Linux: What is the EPEL](https://mangolassi.it/topic/12046/)
* Linux: DNF Package Management

Debian, Ubuntu and the DEB World
* Linux: Installing with APT
Installing from a Tarball
* UNIX: What Is a Tarball

General Administration Tasks

• Linux: Finding What Distro We Are Using
• Linux FAQ: Why Do We Need a Dot Slash Before a Local Command
• Scheduling with cron
• Scheduling with at
• Linux: File Compression Utilities
• Linux: Using tar
• Linux: Zip and 7Zip

Storage and Filesystems

• System Administration: Filesystems
• System Administration: Clustered Filesystems
• Linux: Checking Filesystem Usage with df
• Linux: Directory Utilization with du
  • Linux: Looking for Large Folders with du
  • Linux: Why are df and du Showing Me Different Things?
  • Sparse Files and lastlog
• Linux: Disk Devices
• Linux: Working with Disks, fdisk and parted
• Linux: Creating a Filesystem
• Linux: Mounting Filesystems
• Linux: The Role of the Logical Volume Manager (LVM)
  • Linux: Working with LVM
  • Linux: Extended LVM Reporting Commands
  • Linux Practicum: Adding a Second Storage Drive on CentOS 7 with LVM and XFS
  • Linux Practicum: Removing an LVM Configuration on CentOS 7
• Linux: The Role of MD Software RAID
• Linux: MD Software RAID
• Linux: The Role of DRBD
• Linux: BtrFS
  • Linux Practicum: Adding a Second Storage Device on CentOS 7 with BtrFS
• UNIX: ZFS
• System Administration: Network Filesystems
  • Linux: NFS
    • Linux Practicum: Creating a Simple NFS Server
    • Linux Practicum: Mounting a Simple NFS Share
  • Linux: SMB

Networking

• Linux: Network Bonding and Teaming

Memory and Swap

• Linux: Swap Space and Files
• Linux: Creating Swap
• Linux: Swappiness Behavior

Monitoring and Analysis

• Linux: Using uptime to understand load
• Linux: Using free to view memory usage
• Linux: Using top to view instantaneous performance
• Linux: SAR

System Administration Tasks

• System Administration: Backups

Linux Special Tools

• Linux: Special Tools Outside of the Mainstream
  • Netdata - Local Data Analytics with Cloud Interface
• Linux: Special Tools for Command Line Performance Viewing

Linux Why?

• UNIX: Why Does Root Get a Special Home Directory

BASH Basics

• BASH: if else Constructs
• BASH: case Constructs

Advanced Topics

DRBD

---

• Linux Command References

It's not often that you get messages like this and I wanted to kind of save it somewhere Shout out to @BBigford @Neally @DenisKelley who are all named as well.

https://community.spiceworks.com/topic/1785625-the-influence-of-community-and-personal-inspiration

So we have been kicking around an idea to help turn the NTG Lab up to the next level and we wanted to see if there would be any interesting from the community. The NTG Lab is pretty extensive, and always growing. We get good access to hardware and software from lots of vendors anxious to have us showcase their products to our staff and since we have an isolated lab the availability of licenses to us is generally pretty good (it's kept out of production so vendors know they aren't losing money from a lack of sales.) We have gear such as several enterprise servers from multiple vendors, iSCSI SAN, NAS, hypervisors, etc.

What we are looking to do is move from the small, limited lab facility that we have currently to move into something much more substantial and robust - a full cabinet in an enterprise datacenter.

We are interested in finding out if some community members might be interested in going in for something like $25/mo to get access to the lab. It's not a cloud platform and the system would be a bit information, we aren't looking at hundreds of people here but it would be great to find around twenty. This would be people who know each other, not a system "for sale" and the goal is education, testing and lab - no implication of this being a production system (we have a SAN for goodness sake, this isn't prod!!)

The idea is that this would be a truly incredible system that we can use together for a degree of learning and experimenting that just can't be done in another way. Basically the most amazing home lab you could imagine - hosted in an enterprise datacenter. For the cost of just a few tiny cloud VMs from a cloud provider.

Thoughts?

Everyone learns differently. We all know this, but we rarely take a lot of time to think about what it means practically. Some people are great auditory learners, I know I am. Give me an informative podcast with good information and a long morning walk and I will ingest a lot of great information. Some people need to take notes, others lose everything being said if they take notes. Some people need visual aids. Some need to do things hands on. Some want to investigate and discover while others want to be led and shown. We are all different.

In the working world and especially in fields like IT and software engineering learning is far more than the background practice that we do to get in the door, it is a part of our daily lives. IT is an enormous field and requires that we not only know a lot of foundational background information, but also requires that we constantly stay up to date on products, companies, techniques, tools, trends and more. IT is so large that even if it were to never change or grow we would still be spending our entire careers learning more and more as there is just so much to know!

IT practitioners take learning to a new level that most fields would never suggest or accept. In fact, if you even suggested to people working in most fields that they should have to put in the kind of time that IT professionals much do just to remain current and grow at a moderate pace you would not just be mocked but would often be met with derision. Many people would actually be upset being asked to learn in such a way.

In fields, like IT, where continuous learning are necessities we have to think about learning processes in different ways. For most people in the majority of the world's careers it is adequate that learning be separate from "life and career", that it be set aside as a special case activity only done in special, formal settings such as high school, university, certification training classes and the like. Additional learning or training might be requires throughout a career but it would generally be done is special settings, at scheduled times and be very limited such as a training course for a few days every few years. Very "separate" from normal life.

IT cannot work in this manner effectively. Learning must be constant, it must be part of the lifestyle. We cannot look at learning and education as something we do then set aside, only taking down off of the shelf and dusting off once in a while. We must live it. We must embrace it. It has to be part of every day, every activity. Always learning, always growing, always advancing.

One of the most important skills that any IT practitioner must cultivate is that of being able to teach themselves, constantly. This might be reading books, spending time in deep reflection, doing projects, hanging out on professional social media, reading articles, pursuing certifications, building labs, trialing new products, attending seminars, attending conferences and more. And most likely, doing many or all of these things.

IT requires so much learning, in so many directions, at such a significant pace that there is no reasonable means of approaching the primary corpus of an individual's learning from any formal system. Formal learning systems, classrooms of one type or another, are perfectly reasonable as supplemental learning systems. But these are time consuming, expensive and slow at best and, in most cases, lack even the capability of providing continued support for mid-level career professionals and above. The value of formal systems drops away rapidly as a career progresses and alternative learning methods must be adopted.

IT essentially demands that anyone looking to continue growing in the field, or in reality even those just looking to stand still without losing ground, teach themselves throughout their careers. A formal education process might be useful in kick starting their careers at the onset or help to get them "over a hump" later in life. But by and large self learning is needed at nearly all times.

The reason that I love hiring people who have always used self education and self learning processes while foregoing formal educational processes is not because of a negative associated with those processes, but because the needs of the field demand that self learning be a skill that isn't just passable but is highly honed and natural. If someone has learned through traditional, formal processes then I have to struggle to determine how much passion, determination and practical skill at self learning that person possesses. Why did they choose formal education that is slower, less demanding and more expensive if they enjoy and are good at learning on their own? How do I know that they can continue to grow without needing special resources provided at best, or at worst that they are simply unable to keep growing in IT and will immediately begin the process of failing?

For job candidates that don't demonstrate a strong aptitude, desire and experience in self education I am left with little option but to wonder if they are ready for a career in IT. Of course they may have simply chosen a developmental path that fails to demonstrate some of the most critical industry skills, and that is unfortunate. But candidates that have demonstrated that they can self educate are known to have a critical skill that is needed. That demonstration is one of the best possible factor that I could look for in evaluating a candidate.

Going forward, after hiring someone new, their past knowledge is almost useless. What they have learning is likely outdated, is almost certainly not directly applicable to the work that they will need to be doing and what they need to do will often change anyway and even if they were fully current at the time that they were hired they would still need to be learning regularly just to maintain the level of skill that they had when hired. The skills and experience that someone has when initial hired serves almost exclusively to demonstrate aptitude and interest for the subject material. Of equal or greater importance, but much harder to ascertain, is their ability to learn the new material that will be needed going forward and to do so on their own with minimal additional assistance.

Those that have taught themselves have simply more opportunity to demonstrate their ability to do what is needed to succeed in IT. Which itself is meaningful, it suggests that they understand that requirement of doing so and the value of being able to do so. Candidates who have not done this adequately may themselves not understand what the field will entail once they are working in it and may have no idea how to teach themselves or even if they have the necessary skills and drive to do so anyway. Formal educational processes do so much to avoid these processes that often those who have taken those paths in life may lack the necessary exposure to even answer these questions when presented with them.

Self learning is the best way to prove to employers and to yourself how ready to you are to tackle the ongoing growth and educational demands of the industry.

Jump servers are one of the easier projects to tackle and a great starting place on your Linux journey as they are easy to make, very useful, have little to no external dependencies and are often a foundation point for starting a UNIX infrastructure. In this lab project we will build a very basic Linux Jump Server using CentOS 7.

For a Jump Server we can do a very basic install, we can easily get away with a single vCPU and 1GB of RAM. We need only the most minimal local storage so 8GB should be more than enough. Once we have the settings ready, we can do a minimal install option and accept all of the defaults. No special needs here.

As always we start by patching up to current...

yum -y update
yum -y install epel-release
yum -y install fail2ban

And truly, this is enough to have a starter Jump Box. Very basic, of course, and probably not where we would want to stop. Where do we go from here?

First you would create users and SSH keys and then deploy them to the other boxes that you wish to connect to. This is the core of what makes the Jump Box a Jump Box. This is standard SSH key setup, nothing unique to a Jump Box.

Additional steps that are often interesting are to add two factor authentication to the Jump Box, such as Google Authenticator.

I've been looking for this for years (not knowing the name didn't help) and finally found it. It is a Canadian show that we saw in NY via PBS when I was young. It aired in 1983 and introduced many of the most popular computers of the era (8bit era pre-Mac and pre-Amiga.). I used to watch this with @SonshineAcres and my mom in the years before we owned a computer at home. A very influential, but silly, show for me.

Youtube Video

Youtube Video

People often ask where to start learning Linux, and the answer is always to "just start doing projects." That's hard to do, of course, because someone without server experience might not have a good set of project ideas that would reflect real business processes to start out with. So I am going to start collecting ideas:

• Build a Linux Jump Box
• Build a Linux Logging Server like ELK, Graylog or Splunk
• Build a Linux Monitoring System like Zabbix, Zenoss or Nagio
• Build a Linux Ticketing System like ServiceDesk+ or osTicket
• Build a Linux Documentation Management System like MediaWiki, Dokuwiki or Alfresco.
• Build a Linux VoIP PBX like FreePBX.
• Build a Linux Instant Messaging System like Rocket.Chat, Mattermost or OpenFire.
• Build a Linux Email System like Zimbra.
• Build a Linux Media Server for home movies, music, pictures, etc.
• Build a Linux Backups System.
• Build a Linux Shared Storage System using NFS for other Linux Machines.
• Build a Linux Shared Storage System using Samba for Windows and Mac OSX devices.
• Build a Linux Active Directory server with Samba 4 or FreeIPA.
• Build a DevOps management system with Salt, Ansible, Chef or Puppet.
• Script your builds, script maintenance tasks.

General Project Tips:

• Always use enterprise Linux OS Distros: CentOS / RHEL, OpenSuse / Suse or Ubuntu (mostly in that order)
• Treat the systems as if they were production and secure them, monitor them, etc.
• Always vitualize
• Use real server hardware when you can
• Use enterprise cloud platforms when you can (Amazon, Azure, Rackspace, Softlayer, Digital Ocean, Vultr, etc.)
• Do everything in such a way as to make your current business or any potential employer jealous of your home or lab network. Do this by raising the bar on the home line to be what a business should be like.

Doing this on a Digital Ocean CentOS 7.1 Droplet with 1GB of RAM. Always fully update before starting.

[root@temp-c7-snipeit ~]# cat /etc/redhat-release
CentOS Linux release 7.1.1503 (Core)

setenforce 0
yum -y install epel-release
mkdir -p /var/www/html; cd /var/www/html/
wget https://raw.githubusercontent.com/snipe/snipe-it/master/install.sh && chmod 744 install.sh && ./install.sh

Just accepts the defaults for MariaDB and set the root password. Remember to record this!!

Now a little more, the site should be up but will have errors because we aren't quite done yet...

 cd snipeit
vi app/config/app.php

Now modify the timezone line so that it looks like this...

Now we just have to do one final step to get everything configured and running. Just run this command to configure your first user:

php artisan app:install

Follow the prompts. Once completed, you are done. You can navigate to http://ipaddress/ and you should be able to log in and begin using your Snipe-IT system on CentOS 7 with MariaDB.

It does not take long working in IT to experience this phenomenon. The previous IT professional built what is clearly a disaster and then promptly quit. We see this in small shops where there is a single IT professional that leaves or in shops using an MSP and the MSP is replaced. In both cases the commonality is that when there is a change of IT staff it is a total change of staff - there is a lack of continuity between the old staff and the new staff. What is important here is that there is no one involved in making the old decisions who then is affected by them in the future.

So why do we see this so often? This might seem strange and just a bizarre artefact of IT, but there is a reason for it, I believe. Or reasons, perhaps.

IT is inherently complex and as we well know, it is poorly understood by management and often ignored. Management often sees only the results and not the risk - whether to outages, data loss, audits or whatever. This means that management will often (in fact, almost aways) reward "getting lucky" more than "doing a good job." This creates some very obvious problems. Combine this with the fact that nearly no SMB has a growth plan for IT pros effectively forcing them to change jobs to continue their careers and we have a disastrous combination of factors for the business.

To an employee making big decisions that are only see from the outside and not audited for quality processes there is only results of "it worked" or "it failed" rather than "it works but we are at huge risk of total disaster" and "it didn't work, but nearly did and it wasn't my fault." An employee is, much of the time, more protected by cutting corners than honestly trying to do good work. This is an obvious failing by management to not look for meaningful work and to reward random or worse, reckless, work.

The result is that for best career results in the SMB space much of the time and often in the MSP for the SMB space the best career options can be had by taking reckless chances. In doing this, projects can come in under budget and be completed quickly. Why waste time protecting the environment from disaster, why waste time documenting, why train someone else to take over, why study best practices or common approaches - all of these things add career risk and if the business isn't looking for a "good job" to be done, there is little incentive for the IT staff to consider them as valuable. In reality, even very reckless IT decisions are rarely going to blow up in the first year or two, most disasters happen well down the road. And even very dangerous conditions rarely have a disaster rate higher than 50% over a system lifespan.

For example, using a Walmart bargain bin laptop as a server, no RAID, no backups, with a pirated copy of Windows 2008, without being patched ever still has a better than 50% chance of not creating a disaster for more than two years. Of course we know that this is a terrible businesses decision and represents all kinds of risk to the business, yet if the business is not auditing for good decision making, long term investment thinking, thinking like an owner, risk assessments, legal licensing, industry best practices and so forth, the business would only see "working file server with active directory functionality" and "saving a fortune over other proposed solutions." Of course, the business could fail and lose every bit of their data and be hit by a major Microsoft licensing audit and be totally out of business overnight because of this, but unless those disasters hit the business might never know what risk was taken.

How does a house of cards scenario help the IT pro? In tons of ways:

• It lowers the skill and effort needed to do IT work. This lets marginally skilled IT pros look competent and even, in some cases, like super stars (short term).
• It allows IT to benefit from getting paid full time work, for doing part time effort.
• If the IT pro or MSP can walk away before the disaster hits, they look like heroes and move on to the next job(s) with a sparkling reference that they need only use once or twice before doing the same thing from the next job.
• If the disaster hits before they quit or find another job, the impact is minimal. If the fault for risky house of cards thinking gets pinned on them, they get fired or quit - it's not that much punishment, the risk is low. Find another job, use a different job that didn't have a disaster as a reference. References are a dime a dozen.
• If the disaster hits after moving on, blame the "new guy" for not maintaining the system properly. The same factors that kept management from auditing while you were there likely can't tell if you or the new guy are to blame for the disaster. Heck, he didn't fix the problems so clearly he was okay with how things were, right?
• In easily 60% or more of cases, no disaster will ever happen before systems are retired or the company fails anyway. Most SMBs go out of business in just a few years in general. So the chances that there even will be a failure at all, while insanely high for a business decision where the negative outcomes will often mean bankruptcy, is no big deal to the employee for whom the risks are trivial. So there is a very good chance that several companies that an IT pros does this to will never be any the wiser and will very often thank him for what he did.
• Even if caught, because the business wasn't watching carefully there is a very good chance that the IT Pro can pass blame to equipment, acts of God or whatever. These things happen, just look at all of the other SMBs having unexpected, total disasters. Even when caught, blame is not always assigned.

At the end of the day, IT Pros in the SMB are rarely held accountable in a way that rewards good decision making for the business and rather are more likely to be rewarded, either by the initial company or through subsequent career moves, for having put the business at risk. The risks to the business are huge, while trivial to the IT staff. Conversely, the employee is rarely rewarded for doing a truly excellent job and may be passed over for work if they refuse to cut corners compared to someone offering to "make do with fewer resources and less budget" making it actually a risky career move to do a good job!

At the end of the day, IT cannot fix this problem. Only businesses can. Good business means rewarding employees for good work and holding them accountable to bad work. But if the business rewards high risk scenarios, employees will take that route. Understanding how businesses are incentivizing their own demise or risk is the critical first step to fixing this issue.

@CloudKnight said in Rethinking Virtualization? Start with Hyper-V 2025:

I wouldn't even consider hyper v. Proxmox and xcp-ng are the future

And like, they were the "now and future" like, a decade ago. This isn't a recent thing.

@EddieJennings yeah, that would be nice for sure.

@Oksana LOL, if you are considering Hyper-V in 2025, you should be looking for a different career path.

@Oksana VMware still exists? LOL Why?

@CCWTech Looking at it, I think that it requires a client. Which isn't all bad, but limiting. The convenient web interface is paid only.

While generally not recommended, sometimes we need to back up to a USB drive or other hard mounted backup location in ProxMox.

Problem: USB and other external devices can easily lose their mount while a system is already online causing the mount point to unmount forcefully and leaving a folder where the mountpoint had been. Rarely do we want the backup function of ProxMox to backup to that local folder, we rather want the backup mechanism to fail and then have an opportunity to alert us to fix the backup location issue.

Solution: Inform ProxMox at an application level that the backup location is a mountpoint and not a local folder so that it runs detection prior to backing up.

What to do?

1. Get your list of storage locations from ProxMox
   pvesm status

2. Find your backup location NAME on that list. It is a ProxMox label, not the folder name. For many of you it might just be called something like "backup"

3. Inform ProxMox to test if the location is mounted prior to backing up. Replace the word backup in this example with the NAME you got from step 2.
   pvesm set backup --is_mountpoint yes

4. You are done if that succeeds silently, but you can double check that the change has been made by looking for it in this file:
   cat /etc/pve/storage.cfg

We have Tactical, it just automates MEshCentral poorly for Windows and doesn't work for non-Windows at all. We actually have to replace it because of that.

If you haven't played with it for tiny scale stuff, TailScale makes this SO easy to do.

@Mario-Jakovina you are correct, in bridge mode a telecom router SHOULD be essentially transparent. It's just an extra switch at that point.

MT is good. I always prefer a dedicated router/firewall. Then my APs separate.

If you run Ubuntu or other Linux system using the UFW firewall mechanism, and you probably want to limit at least some ports to only receiving traffic from CloudFlare's proxy servers. CloudFlare provides scripts for iptables, but not for UFW. But they do provide their IP list in a handy format. So here is all you need to do...

cd /tmp
wget https://www.cloudflare.com/ips-v4 -O ips-v4-$$.tmp
wget https://www.cloudflare.com/ips-v6 -O ips-v6-$$.tmp

for cfip in `cat ips-v4-$$.tmp`; do echo "ufw allow from $cfip to any port 80 proto tcp"; done
for cfip in `cat ips-v6-$$.tmp`; do echo "ufw allow from $cfip to any port 80 proto tcp"; done```

Notice this doesn't take action, it produces a handy human readable set of ufw instructions that you can audit before running. Just copy the output to a file and "bash file" to take action. Make sure to set the port to 443 in most cases, or run twice, once with 80 and once with 443.