34-35.linux-basics.md

Getting Comfortable with the Linux Operating System

Note: The explanation of Linux commands in this lesson is based on a Linux distribution named CentOS. I used an Alibaba Cloud server, and the system version was CentOS Linux release 7.6.1810. Different Linux distributions have some small differences in shell commands and utility programs, but these differences are very small.

History of Operating Systems

A computer system with hardware only and no software is called a "bare machine". It is hard for us to use a bare machine to do daily computer work such as storage and computing, so we must use special software to control the hardware. The software closest to computer hardware is system software, and the most important part of that is the operating system. An operating system is a collection of programs that controls and manages all computer hardware and software resources, carries out resource allocation and task scheduling, and provides interfaces and an environment for system users and other software.

No Operating System (Manual Operation)

In the early days, before operating systems existed, people first loaded a program paper tape or card into the computer, then started the input device to send the program into the computer, and then started the program by using the switches on the console. When the program finished running, the printer printed the result, and the user unloaded the paper tape or cards and took them away. Then the second user used the computer and repeated the same steps. During the whole process, the user used the machine alone, the CPU waited for manual operation, and resource usage was very low.

Batch Systems

First, start a supervisory program on the computer. Under the control of this supervisory program, the computer can automatically and in batches process one or more users' jobs. After one batch of jobs is finished, the supervisory program reads jobs again from the input device and stores them in the tape drive. Then it repeats the same steps to process tasks. The supervisory program keeps processing jobs and makes automatic job handoff happen, which reduces job setup time and manual operation time, and improves the use of computer resources. Batch systems can be divided into single-program batch systems, multiprogram batch systems, online batch systems, and offline batch systems.

Time-Sharing Systems and Real-Time Systems

A time-sharing system divides the processor running time into very short time slices, and gives the processor to each online job in turn according to the time slice. If a job cannot finish its computation within its time slice, the job is paused for a while, the processor is given to another job, and the original job waits for the next scheduling round to continue running. Because the computer is fast and jobs take turns very quickly, each user feels that they have one computer to themselves. Each user can send different operation control commands to the system through their own terminal, and finish tasks in a full human-computer interactive environment. To solve the problem that time-sharing systems could not respond to user commands in time, real-time systems appeared. These systems can finish event processing within a strict time range and respond in time to random outside events.

General-Purpose Operating Systems

1. In the 1960s, IBM's System/360 series had a unified operating system called OS/360.

2. In 1965, AT&T's Bell Labs joined GE and MIT in the MULTICS project.

3. In 1969, the MULTICS project failed. Ken Thompson was staying at home, and for playing the game Space Travel, he used assembly language to develop Unics on the already outdated PDP-7.

   Note: It is hard to imagine that such a great operating system as Unix was developed by a programmer staying at home, on an outdated machine, just for playing a game.

4. In 1970 and 1971, Ken Thompson and Dennis Ritchie rewrote Unics in the B language on the PDP-11, and at Brian Kernighan's suggestion renamed it Unix.

   

5. In 1972 and 1973, Dennis Ritchie invented the C language to replace the less portable B language and began rewriting Unix in C.

6. In 1974, Unix released the milestone Version 5, which was almost entirely implemented in C.

7. In 1979, starting from Unix Version 7, AT&T privatized Unix through new licensing terms.

8. In 1987, Professor Andrew S. Tanenbaum wanted to explain the details of operating-system internals to students in class, so to avoid copyright disputes, he decided to develop a Unix-compatible operating system himself without using any AT&T source code. This system was named Minix.

   

9. In 1991, while Linus Torvalds was studying at the University of Helsinki in Finland, he tried to do some development work on Minix. But because Minix was only an operating system for teaching and was not very powerful, Linus wrote a disk driver and file system so that he could more easily read, write, and download files in the university's newsgroup and mail system. These things formed the early shape of the Linux system kernel.

   

The picture below is the map of the Unix operating-system family.

Linux Overview

Linux is a general-purpose operating system. An operating system is responsible for task scheduling, memory allocation, and handling peripheral device I/O. An operating system is usually made up of two parts: the kernel (the core program that runs other programs and manages hardware devices such as disks and printers) and system programs (device drivers, low-level libraries, shell, service programs, and so on).

The Linux kernel was developed by Linus Torvalds and was released in September 1991. As a product of the internet age, the Linux operating system was developed together by many developers around the world, and it is a free operating system. Note that freedom and free of charge are not the same idea.

Advantages of Linux

1. It is a general-purpose operating system and is not tied to specific hardware.
2. It is written in C, has strong portability, and has kernel programming interfaces.
3. It supports multiple users and multiple tasks, and supports a safe layered file system.
4. It has many practical programs, complete network functions, and strong support documents.
5. It has reliable security and good stability, and is more friendly to developers.

Linux Distribution Versions

1. Red Hat
2. Ubuntu
3. CentOS
4. Fedora
5. Debian
6. openSUSE

Basic Commands

Linux commands in Linux systems usually use the format shown below.

command [named options] [target]

1. Get login information: w / who / last / lastb

   [root ~]# w
    23:31:16 up 12:16,  2 users,  load average: 0.00, 0.01, 0.05
   USER     TTY      FROM             LOGIN@   IDLE   JCPU   PCPU WHAT
   root     pts/0    182.139.66.250   23:03    4.00s  0.02s  0.00s w
   jackfrue pts/1    182.139.66.250   23:26    3:56   0.00s  0.00s -bash
   [root ~]# who
   root     pts/0        2018-04-12 23:03 (182.139.66.250)
   jackfrued pts/1        2018-04-12 23:26 (182.139.66.250)
   [root ~]# last
   root     pts/0        117.136.63.184   Sun May 26 18:57   still logged in
   reboot   system boot  3.10.0-957.10.1. Mon May 27 02:52 - 19:10  (-7:-42)
   ...

2. Check the shell you are using: ps

   A shell is also called a shell program. It is the translator between the user and the operating-system kernel. Put simply, it is the interface for interaction between people and the computer. In many Linux systems, the default shell is bash (Bourne Again SHell), because it can use the Tab key to complete commands and paths, can save command history, can conveniently configure environment variables, and can execute batch operations.

3. View the description and location of a command: whatis / which / whereis

   [root ~]# whatis ps
   ps (1)        - report a snapshot of the current processes.
   [root ~]# whereis ps
   ps: /usr/bin/ps /usr/share/man/man1/ps.1.gz
   [root ~]# which python
   /usr/bin/python

4. Clear what is shown on the screen: clear

5. View help documents: man / info / --help / apropos

   [root@izwz97tbgo9lkabnat2lo8z ~]# ps --help
   Usage:
    ps [options]
   ...
   [root@izwz97tbgo9lkabnat2lo8z ~]# man ps
   PS(1)                                User Commands                                PS(1)
   NAME
          ps - report a snapshot of the current processes.
   ...

6. View the system and host name: uname / hostname

   [root@izwz97tbgo9lkabnat2lo8z ~]# uname
   Linux
   [root@izwz97tbgo9lkabnat2lo8z ~]# hostname
   izwz97tbgo9lkabnat2lo8z
   [root@iZwz97tbgo9lkabnat2lo8Z ~]# cat /etc/centos-release
   CentOS Linux release 7.6.1810 (Core)

7. Time and date: date / cal

   [root@iZwz97tbgo9lkabnat2lo8Z ~]# date
   Wed Jun 20 12:53:19 CST 2018
   [root@iZwz97tbgo9lkabnat2lo8Z ~]# cal
         June 2018
   Su Mo Tu We Th Fr Sa
                   1  2
    3  4  5  6  7  8  9
   ...

8. Restart and shut down: reboot / shutdown

   [root ~]# shutdown -h +5
   Shutdown scheduled for Sun 2019-05-26 19:34:27 CST, use 'shutdown -c' to cancel.
   ...
   [root ~]# shutdown -r 23:58
   Shutdown scheduled for Sun 2019-05-26 23:58:00 CST, use 'shutdown -c' to cancel.

9. Exit login: exit / logout

10. View command history: history

[root@iZwz97tbgo9lkabnat2lo8Z ~]# history
...
452  ls
453  cd Python-3.6.5/
454  clear
455  history
[root@iZwz97tbgo9lkabnat2lo8Z ~]# !454

Utility Programs

File and Directory Operations

1. Create and delete empty directories: mkdir / rmdir

2. Create and delete files: touch / rm

   • touch is used to create an empty file or change file times. In Linux, a file has three times:
     • time when content is changed: mtime
     • time when permissions are changed: ctime
     • last access time: atime
   • Important options of rm:
     • -i: interactive delete. It asks about every item.
     • -r: delete directories and recursively delete files and directories in them.
     • -f: force delete. Ignore files that do not exist and show no prompt.

3. Switch and view the current working directory: cd / pwd

   Note: The cd command can be followed by a relative path (using the current path as the reference) or an absolute path (starting with /) to switch to the specified directory. You can also use cd .. to go back to the upper-level directory.

4. View directory contents: ls

   • -l: view files and directories in long format
   • -a: show files and directories that start with a dot (hidden files)
   • -R: when there is a directory, keep expanding it recursively
   • -d: list only directories, not other content
   • -S / -t: sort by size / time

5. View file content: cat / tac / head / tail / more / less / rev / od

   Note: The command wget is used above. It is a network downloader program that can download resources from a specified URL.

6. Copy and move files: cp / mv

7. Rename files: rename

8. Find files and search content: find / grep

   Note: grep can use regular expressions when it searches strings. If you need regular expressions, you can use grep -E or directly use egrep.

9. Create links and view links: ln / readlink

   [root@iZwz97tbgo9lkabnat2lo8Z ~]# ls -l sohu.html
   -rw-r--r-- 1 root root 212131 Jun 20 19:15 sohu.html
   [root@iZwz97tbgo9lkabnat2lo8Z ~]# ln -s /etc/centos-release sysinfo
   [root@iZwz97tbgo9lkabnat2lo8Z ~]# cat sysinfo
   CentOS Linux release 7.4.1708 (Core)

   Note: Links can be divided into hard links and soft links (symbolic links). A hard link can be thought of as a pointer to file data, like the reference count of an object in Python. Each time one hard link is added, the file's link count increases by 1. Only when the link count becomes 0 can the storage space of that file possibly be covered by other files. When we usually delete a file, we are not really deleting the data on the hard disk. We only delete a pointer, or we can say one usage record of the data. A soft link is similar to a shortcut in Windows. When the file linked by the soft link is deleted, the soft link also becomes invalid.

10. Compress and decompress, and archive and unarchive: gzip / gunzip / xz

[root@iZwz97tbgo9lkabnat2lo8Z ~]# wget http://download.redis.io/releases/redis-4.0.10.tar.gz
...
[root@iZwz97tbgo9lkabnat2lo8Z ~]# gunzip redis-4.0.10.tar.gz
[root@iZwz97tbgo9lkabnat2lo8Z ~]# ls redis*
redis-4.0.10.tar

11. Archive and unarchive: tar

[root@iZwz97tbgo9lkabnat2lo8Z ~]# tar -xvf redis-4.0.10.tar
redis-4.0.10/
redis-4.0.10/.gitignore
...

Note: Both archiving (also called creating an archive) and unarchiving use the tar command. Usually, creating an archive needs the three options -cvf. Here, c means create, v means show verbose details, and f means the file of the archive. Unarchiving needs -xvf, where x means extract, and the other two options are the same as for creating an archive.

12. Turn standard input into command-line arguments: xargs

This command is often used in pipes and redirection.

13. Show the file name or directory name in a path: basename / dirname
14. Other related tools:

• sort: sort content
• uniq: remove adjacent duplicates
• tr: replace specified content with new content
• cut / paste: cut and paste content columns
• split: split files
• file: determine file type
• wc: count lines, words, and bytes
• iconv: convert encodings

[root ~]# cat foo.txt
grape
apple
pitaya
[root ~]# cat bar.txt
100
200
300
400
[root ~]# paste foo.txt bar.txt
grape   100
apple   200
pitaya  300
        400
...

Pipes and Redirection

1. Using a pipe, |

   Example: count the number of files under the current directory.

   find ./ | wc -l

   Example: list files and directories under the current path and add a number to each item.

   ls | cat -n

   Example: count records in record.log that contain AAA but do not contain BBB.

   cat record.log | grep AAA | grep -v BBB | wc -l

2. Output and error redirection: > / >> / 2>

3. Input redirection: <

4. Multiple redirection: tee

   The following command both shows the result of ls on the terminal and appends it to ls.txt.

   ls | tee -a ls.txt

Aliases

1. alias
2. unalias

Aliases are convenient for mapping commonly used command combinations to short names.

[root ~]# alias ll='ls -l'
[root ~]# alias frm='rm -rf'
[root ~]# unalias frm

Text Processing

1. Stream editor: sed

   sed is a tool for operating on, filtering, and transforming text. It can append lines, insert lines, delete lines, and do substitutions. Like many commands shown earlier, it does not modify the original file unless you explicitly redirect or save the result.

   [root ~]# cat -n fruit.txt
        1  banana
        2  grape
        3  apple
        4  watermelon
        5  orange
   [root ~]# sed '2a pitaya' fruit.txt
   banana
   grape
   pitaya
   apple
   watermelon
   orange

2. Pattern scanning and processing language: awk

   awk is both a programming language and one of the most powerful text-processing tools in Linux. It can extract specific columns from text, use regular expressions to retrieve content, display specified rows, and perform statistics and calculations. In short, it is extremely powerful.

   [root ~]# cat fruit2.txt
   1       banana      120
   2       grape       500
   3       apple       1230
   4       watermelon  80
   5       orange      400
   [root ~]# awk 'NR==3' fruit2.txt
   3       apple       1230
   [root ~]# awk '{print $2}' fruit2.txt
   banana
   grape
   apple
   watermelon
   orange

User Management

1. Create and delete users: useradd / userdel

   • -d: specify the user's home directory when creating a user
   • -g: specify the user's primary group when creating a user

2. Create and delete user groups: groupadd / groupdel

   Note: User groups mainly exist to make it easier to manage a set of users together.

3. Change passwords: passwd

   Note: If passwd is used without specifying a target, it changes the current user's password. If you need to modify many users' passwords in bulk, you can use chpasswd.

   • -l / -u: lock or unlock a user
   • -d: clear a user's password
   • -e: force the password to expire immediately, so the user must change it at login
   • -i: disable the user some number of days after password expiration

4. View and modify password validity periods: chage

5. Switch users: su

6. Execute commands as an administrator: sudo

   Note: If a user is expected to run commands with administrator privileges, that user must appear in the sudoers list. The sudoers file lives under /etc.

7. Edit the sudoers file: visudo

8. Display user and group information: id

9. Send messages to other users: write / wall

10. View or set whether you accept messages from other users: mesg

File System

Files and Paths

1. Naming rules: the maximum file-name length depends on the file-system type, but generally a file name should not exceed 255 characters. Although most characters can be used in file names, it is best to stick to upper- and lower-case English letters, digits, underscores, and periods. File names can contain spaces, but it is better to avoid them, because then you need either quotes or escaping.
2. Extensions: file extensions are optional in Linux, but they help us understand file contents. Some applications recognize files by their extensions, but many do not. Commands such as file do not rely on extensions.
3. Hidden files: files whose names begin with a dot are hidden files in Linux.

Directory Structure

1. /bin - binary files for basic commands
2. /boot - static files for the boot loader
3. /dev - device files
4. /etc - configuration files
5. /home - parent directory for ordinary users' home directories
6. /lib - shared library files
7. /lib64 - shared 64-bit library files
8. /lost+found - stores unlinked files
9. /media - mount points for automatically detected devices
10. /mnt - temporary mount points
11. /opt - installation location for optional add-on packages
12. /proc - kernel and process information
13. /root - the home directory of the superuser
14. /run - runtime data required by the system
15. /sbin - binaries for the superuser
16. /sys - a pseudo file system for devices
17. /tmp - temporary files
18. /usr - user applications
19. /var - variable data

Access Permissions

1. chmod - change file mode bits

   Note: chmod can change permissions in two ways: symbolic notation and numeric notation. Besides chmod, umask can be used to specify which permission bits will be removed from the default permissions of new files.

2. chown - change file owner

3. chgrp - change group ownership

Disk Management

1. List file-system disk-usage status: df
2. Disk partition-table operations: fdisk
3. Disk partitioning tool: parted
4. Format a file system: mkfs
5. File-system checking: fsck
6. Convert or copy files: dd
7. Mount and unmount: mount / umount
8. Create, activate, and disable swap partitions: mkswap / swapon / swapoff

Note: Running the commands above carries some risk. If you do not understand them clearly, do not use them casually. When using them, it is best to follow reference materials carefully and confirm whether you really intend to perform the operation.

Editor: vim

1. Start vim. You can launch it with either vi or vim. A file name can be specified at launch, or you can name the file when saving.

2. Command mode, insert mode, and last-line mode. vim starts in command mode. Pressing i enters insert mode. Pressing Esc returns to command mode. Entering : from command mode enters last-line mode, where q! forces quit without saving. In command mode, v enters visual mode, where a region can be selected and operated on.

3. Save and quit: :wq, q!, ZZ, :w

4. Cursor operations

   • h, j, k, l
   • Ctrl+y, Ctrl+e
   • Ctrl+f, Ctrl+b
   • G, gg, and numeric line jumps

5. Text operations

   • delete: dd, d$, d0, dw, :%d
   • copy and paste: yy, p
   • undo and redo: u, Ctrl+r
   • sort content: %!sort

6. Search and replace

   • searching with /
   • replacing with commands such as :1,$s/doc.*/hello/gice

7. Settings in last-line mode

   • set ts=4
   • set nu / set nonu
   • syntax on / syntax off
   • set ruler
   • set hls / set nohls

   Note: If you want these settings to take effect every time vim starts, write them to .vimrc in the user's home directory.

8. Advanced tricks

   • compare multiple files: vim -d foo.txt bar.txt
   • open multiple files
   • split windows with sp or vs
   • map shortcut keys with map or inoremap
   • record and replay macros with q, registers, and @

Software Installation and Configuration

Using Package Managers

1. yum, Yellowdog Updater Modified
   • yum search
   • yum list installed
   • yum install
   • yum remove
   • yum update
   • yum check-update
   • yum info
2. rpm, Red Hat Package Manager
   • install packages: rpm -ivh <packagename>.rpm
   • remove packages: rpm -e <packagename>
   • query packages: rpm -qa

The lesson uses Nginx as an example of installation through yum, and MySQL as an example of installing RPM packages manually. It also notes that MySQL and MariaDB have conflicting lower-level dependencies, which is why some MariaDB libraries may need to be removed first.

Download, Extract, and Configure Environment Variables

The lesson uses MongoDB as an example here. The rough process is to download the package, extract it, add its bin directory to PATH, reload the shell configuration, and then verify the installed binaries with version commands.

Build and Install from Source

1. Installing Python 3.6
2. Installing Redis 3.2.12

Note: After installing software such as Python, it is often still necessary to update PATH, either in .bash_profile under the user's home directory or in /etc/profile. The former behaves like a user-level environment variable, while the latter acts like a system-wide environment variable.

Configuring Services

Under Linux we can install and configure many kinds of services, turning the system into a database server, web server, cache server, file server, message-queue server, and so on. Most Linux services are configured as daemons, background processes that continue to run without preventing Linux from shutting down cleanly. That is why many service names end with the letter d, short for daemon, such as firewalld, mysqld, and httpd. After a service is installed, it can usually be started, stopped, and otherwise managed with systemctl or service.

Examples from the lesson:

1. Start the firewall service: systemctl start firewalld
2. Stop the firewall service: systemctl stop firewalld
3. Restart the firewall service: systemctl restart firewalld
4. View firewall status: systemctl status firewalld
5. Enable or disable the firewall at boot: systemctl enable firewalld / systemctl disable firewalld

Scheduled Tasks

1. Execute commands at a specified time

   • at: queue a task to run at a specified time
   • atq: view the pending task queue
   • atrm: remove a queued task

2. Crontab task scheduling

   crontab -e opens the crontab for editing and lets you configure scheduled jobs with Cron expressions. The lesson also points out that related cron files live under /etc, and the system crontab file itself includes comments explaining the Cron fields.

Network Access and Management

1. Secure remote connection: ssh
2. Fetch resources over the network: wget
3. Send and receive mail: mail
4. Older network configuration tool: ifconfig
5. Newer network configuration tool: ip
6. Network reachability checks: ping
7. Display or manage route tables: route
8. View network services and ports: netstat / ss
9. Capture and inspect traffic: tcpdump
10. Secure file copy: scp
11. File synchronization: rsync
12. Secure file transfer: sftp

Process Management

1. View processes: ps
2. Display the process tree: pstree
3. Find processes matching given conditions: pgrep
4. Terminate a process by PID: kill
5. Terminate a process by name: killall / pkill
6. Put a process in the background
   • Ctrl+Z
   • &
7. Query background jobs: jobs
8. Continue a stopped job in the background: bg
9. Bring a background job to the foreground: fg
10. Adjust runtime priority: nice / renice
11. Keep a process running after logout: nohup
12. Trace system calls: strace
13. View the current runlevel: runlevel
14. Monitor resource usage in real time: top

System Diagnostics

1. Diagnose startup issues: dmesg
2. View system activity information: sar
3. View memory usage: free
4. Virtual-memory statistics: vmstat
5. CPU statistics: mpstat
6. View a process's memory map: pmap
7. Report device CPU and I/O statistics: iostat
8. Display all PCI devices: lspci
9. Display the status of inter-process communication facilities: ipcs

Shell Programming

As mentioned earlier, the shell is the application that connects the user and the operating system. It provides the interface for human-computer interaction. Shell scripts are programs written for the shell. We can use them for system management and file operations. In short, writing shell scripts is a basic skill for anyone who uses Linux seriously.

The internet contains a great deal of shell-scripting material, so this lesson does not attempt a complete treatment. Instead, it gives several small examples:

1. Read two integers, m and n, and calculate the sum of all integers from m to n.
2. Automatically create a folder and a specified number of files.
3. Automatically install a specified version of Redis.

Related Resources

1. Common Linux command-line shortcuts

   Shortcut	Description
   tab	autocomplete a command or path
   Ctrl+a	move the cursor to the start of the line
   Ctrl+e	move the cursor to the end of the line
   Ctrl+f	move the cursor one character to the right
   Ctrl+b	move the cursor one character to the left
   Ctrl+k	cut from the cursor to the end of the line
   Ctrl+u	cut from the cursor to the beginning of the line
   Ctrl+w	cut the word before the cursor
   Ctrl+y	paste the previously cut content
   Ctrl+c	interrupt the currently running task
   Ctrl+h	delete the character before the cursor
   Ctrl+d	exit the current shell
   Ctrl+r	search command history
   Ctrl+g	exit history search
   Ctrl+l	clear the screen and open a new line at the top
   Ctrl+s	lock the terminal temporarily
   Ctrl+q	unlock the terminal
   Ctrl+z	stop the current task and put it in the background
   !!	execute the previous command
   !number	execute the history command with that number
   !letter	execute the most recent command starting with that letter
   !$ / Esc+.	retrieve the last argument of the previous command
   Esc+b	move to the beginning of the current word
   Esc+f	move to the end of the current word

2. What the sections of a man page mean

   Section	Description
   NAME	the command's name and short introduction
   SYNOPSIS	the basic syntax of the command
   DESCRIPTION	a detailed description of the command and its options
   OPTIONS	explanations of command-line options
   EXAMPLES	reference usage examples
   EXIT STATUS	exit codes; typically 0 means success
   SEE ALSO	other related commands or information
   BUGS	known issues related to the command
   AUTHOR	information about the author