Unix

1. Unix Programmer's Manual -- The book that is recommended to read along.

2. echo prints what users type on the screen. When typing a password, echo is turned off.

3. CONTROL == CTL / CNTL / CTRL

   1. RETURN == ctl-m
   2. ctl-d means no more input
   3. ctl-g rings the bell on the terminal
   4. ctl-h is backspace
   5. ctl-i is tab

4. DELETE == RUBOUT / BREAK / INTERRUPT

5. DELETE stops a program immediately by pressing ctl-c

6. Case distinctions matter, use lower case

7. The prompt in the shell uses a dollar sign or a percent sign.

   1. $ means the user is non-root

8. If each letter is typed twice in the terminal, which is strange, you can read the commmand stty (set terminal options) in Section 1 of the Manual

   1. Some UNIX commands:
      1. who -- list users that are using the machine
      2. who am i shows more details than whoami
      3. stty -tabs -- the system convert tabs into the right number of spaces

9. For creating files: type vi / emacs / ed to use text editors

10. ls list names of all files in the current folder

    1. ls -t / ls -l / ls -lt file-a / ls -u / ls -r

11. For printing files, cat / pr

12. For editing files, mv / cp / rm

13. wc file-a outputs： 12 34 546 file-a , 12 is line number, 34 is word number and 546 is character number. (line-word-chars)

14. grep abc file-a shows the line that matches abc in file-a, the search is line-by-line

15. grep -v abc file-a shows the lines that don't match abc in file-a (opposite to the above )

16. grep xxxx file-a file-b can search several files at the same time

17. sort file-a, the sorting is line-by-line

    1. sort -r reverse normal order

    2. sort -n numeric order

    3. sort -nr reverse numeric order

    4. sort -f perform case-insensitive sorting

    5. sort -u remove duplicate lines before sorting

    6. sort file-a -o file-b write sorted output to file-b instead of dispaying it on the terminal

18. tail -- last 10 lines of the file

    1. tail -n 2 file-a / tail -2 file-a print the last two lines of the file

    2. tail +2 file-a prints the file starting from the second line

    3. tail -f (follow) output appended data as file grows (used for monitoring logs or viewing real-time updates)

    4. tail -q file-a file-b (quiet) suppress header info when viewing multi files

    5. tail -c 100 file-a display last 100 bytes of the file

19. cmp compare two files byte by byte, return the first byte position at which the files differ

    1. cmp -b a.txt b.txt print differing bytes in octal

    2. cmp -i 100 a b ignore the first 100 bytes of both files

    3. cmp -l == cmp --verbose

    4. cmp -s == cmp --quiet / cmp --silent

20. diff a.txt b.txt compares two files line by line and display differences between two files, shows what actions(add/delete) should be taken to make file-a become file-b

    1. -u / --unified displays in a more user-friendly way

    2. -c / --context displays in a different way which provides additional context around the lines

    3. -r / --recursiveshows difference of files and sub-folders(depth-1) in two folders, if there are same file names in two folders, then the content of files will be differed line by line.

    4. -q/--brief return xx and xx differ if two files differ, otherwise return None

    5. -i/--ignore-case perform a case-insensitive comparison

    6. -w / --ignore-all-space ignore all whitespace differences

21. sed is stream editor, which changes the output of a file but does not change the file itself.

    1. sed 's/abc/new_abc/' file-a replace the first abc with new_abc of each line in file a

    2. sed 's/abc/new_abc/2' file-a replace the second abc with new_abc of each line in file a

    3. sed 's/abc/new_abc/g' file-a replace all the abc with new_abc of each line in file a

    4. sed 's_abc_abcde_2g' file-a replace all the abc with abcde since the second abc of each line in file a

    5. sed 's/abc/{&}/g' file-a replace all abc with {abc} in file a

    6. sed 's/abc/{& &}/g' file-a replace all abc with {abc abc} in file a

    7. sed '3 s/abc/new_abc/' file-a replace the first abc with new_abc of the 3rd line in file a

    8. sed '2 d' file-a delete the 2nd line of file a

    9. sed '5,$ d' file-a delete the fifth line of file a to the end (only the first four lines of file a will be the output, file-a has no change)

    10. grep abc file-a == sed -n '/abc/ p' file-a

    11. grep -v abc file-a == sed -n '/abc/ !p' file-a

    12. sed '/unix/ i "Add a new line"' file-a insert a line "Add a new line" before there is unix in each line of file-a

    13. sed '/unix/ c "Add a new line"' file-a replace the line of file-a with "Add a new line" when there is unix in it

    14. sed 'y/ul/UL' file-a replace all the lowercase of u and l with U and L in file-a

22. awk can output a file using some special characters to make columns

    1. awk '{print $1}' file-a output the first column of file-a (the default delimiter is space)

    2. awk -F"," '{print $1}' file-a print the first col of file-a using "," as the delimiter

    3. awk -F"," '{print $1, $4, $7}' file-a print the 1st, 4th, 7th column of file-a using "," as the delimiter

    4. awk -F"," 'BEGIN{sum=0}{sum=sum+$5;print sum}' file-a add up the sum of the fifth col of file-a and prints the sum on every line (when the 5th col of the line is not a number or there is no 5th col in the line, then the value of the 5th col will be zero by default)

    5. awk -F"," 'BEGIN{sum=0;counter=0}{sum=sum+$5;counter=counter+1;print sum,counter}' file-a prints sum and counter of each line in file-a

    6. awk -F"," 'BEGIN{sum=0}{sum=sum+$5}END{print sum}' file-a print out the final sum of adding up all the value of the fifth col of the each line in file-a

    7. awk -F"," 'BEGIN{sum=0;counter=0}{sum=sum+$5;counter=counter+1;print sum,counter}' file-a

    8. awk -F"," 'BEGIN{sum=0;counter=0}{sum=sum+$5;counter=counter+1}END{print sum/counter}' file-a prints out the average value of 5th col of each line in file-a

    9. if the first line of a file is actually a table header, then you can set counter=-1 in the above command

23. Filename shorthand is a service of the shell. If you want to turn off the special meaning of "*,?" etc, then enclose them in single quotes or precede the character with a backslash.

    1. ls '?'

    2. ls \*

24. Difference between ls * and echo *

    ls * list all the files and folders in the directory with a line break for each item, while echo * prints out all the names of files and folders continously without any space or line breaks

25. Difference between ls / and echo /

    ls / list all folders in the directory while echo / only prints out "/"

26. cp file-a file-b > file-c copy all the contents of file a and b into file c by overriding file c

27. cp file-a file-b >> file-c copy all the contents of file a and b into file c by appending it to the end of file c

28. sort < file-a sort the contents in file-a

29. who;date execute commands in sequence

30. who && date execute commands in sequence after the first command runs successfully, if not, prints out error and then stop.

31. top & run the command in the background

32. stty list settings for the shell

33. Some shell variables: $PS1 represents the primary prompt string; $PATH , $HOME, $TERM

34. Use abbreviation for variables is ok, for example, d=$HOME;echo $d

35. Everything in UNIX is a file.

    1. File is a sequence of bytes.

    2. A byte is 8 bits long.

36. 
    $ ls -l junk
    
    -rw-r--r-- 1 you 36 Sep 27 06:11 junk
    
    $
    

    From above, junk is a file with 36 bytes created by you.

37. od file-b or od -c file-a prints out all the bytes in a file.(-c means interpret bytes as characters) (od is octal dump)

38. od -x file-a prints in hex, -d in decimal.

39. After each line, there is the ascii newline character, which is 012 in octal value.

    1. 012 is \n new line in octal value

    2. 010 is \b backspace

    3. 011 is \t tab

    4. 015 is \r carriage return

40. The kernel echoes RETURN as a carriage return and a newline, but stores only the newline in the input.

41. "CRLF" is carriage return and linefeed.

42. There is no end of file character in UNIX files. The kernel keeps track of file lengths.

43. When read is called, "zero byets being returned" means end-of-file.

44. ctl-d sends what you have typed to the terminal, if you typed nothing, then it looks like the end of file.

45. The types of files does not be determined by names, but by the certain binary number written in the beginning of files.

46. The current directory is an attribute of a process.

47. du disk usage -- how much disc space is consumed by files in a folder, including all its subdirectories.

48. root is the special user that carries super-user privileges.

49. ls -l file-a shows the permissions info of file a, ls -lg xxx shows the group of the user instead of user name.

50. File permissions: rwx, file type: - / d

51. 
    ls -l /bin/passwd
    
    -rwsr-xr-x 1 root xxxxxxxx
    

    sr means when the file is executed, the current user is given permissions to eit the file. In this case, any user can run the passwd command to change its own password.

    sr is "set-uid" to root. It's a great idea to solve security problems. For example, updating a game program for players. But it needs the "set-uid" file to be correct. And the permissions of the file should be

    1. "only root user has 'write' access"

    2. "everybody has 'execute' access"

52. For chmod, 4 for read, 2 for write and 1 for execute permission.

    1. chmod +x command-a turn on a permission

    2. chmod -x command-a turn off a permission

53. A file has several components:

    1. name

    2. contents

    3. admin info, such permissions and modification times -- stored in "i-node"

    4. In inode, there are three times:

       1. last modified of the file

       2. last used of the file

       3. the time that the inode itself was last changed (eg. set permissions)

    5. ls -i file-a shows the i-number in decimal

54. ln is to give more than one names to the same file. Their i-number is the same.

55. df disk free space -- shows available space on the mounted file subsystems.

56. tee file-a both prints output and writes output to file-a, eg. who | tee temp.txt

57. Commands

    1. simplest command is a single word

    2. a command ends with a newline or a semicolon ";" or ampersand "&"

    3. date;who | wc is different from (date;who) | wc

    4. sleep 5 && date is different from sleep 5 & date

    5. (sleep 5; echo tea is ready) &

    6. cat file-a | grep abc & == (cat file-a | grep abc) &

58. Command arguments and shell metacharacters

    1. <, >, |, ; and & are not arguments

    2. so are the characters that are used as regex, eg. *, ?

    3. double quotes the special characters to prevent them being interpreted

    4. so does a backslash \ do the job

    5. echo "***" == echo \*\*\*

    6. p1 && p2 run p1; if successful, run p2

    7. p1 || p2 run p1; if unsuccessful, run p2

    8. echo $(p1) execute command p1

    9. echo */* prints all files and folders in depth 1 and 2

    10. grep "^-" file-a find the line that begins with "-" in file a

    11. touch "\a" create a file with name "\a"

59. create new commands

    
       $ echo 'date' > a.txt; sh a.txt  // single or double quotes will work
    
       Thu Jun  1 01:12:14 EDT 2023
    
       $ sh < a.txt
    
       Thu Jun  1 01:12:14 EDT 2023
    
       $ chmod 755 a.txt; ./a.txt
    
       Thu Jun  1 01:12:14 EDT 2023
    

$ echo "chmod +x $1" > cx; sh cx cx   // use single quotes here, double quotes won't work

$ ./cx a.txt; ./a.txt

Thu Jun  1 01:12:14 EDT 2023

5. The commands above can change to $ echo 'chmod +x $*' > cx so that it can take up different numbers of arguments. So does the command echo 'wc -l $*' > lc.

6. $* means you can take up flexible number of variables and args.

7. echo grep "$*" file-a here, because $* is double quoted