Linux Shell Scripting Cookbook

Table of Contents Preface 1 Chapter 1: Shell Something Out 7 Introduction 7 Printing in the terminal 9 Playing with variables and environment variables 12 Doing math calculations with the shell 17 Playing with file descriptors and redirection 19 Arrays and associative arrays 25 Visiting aliases 27 Grabbing information about terminal 29 Getting, setting dates, and delays 30 Debugging the script 33 Functions and arguments 35 Reading the output of a sequence of commands 38 Reading "n" characters without pressing Return 40 Field separators and iterators 41 Comparisons and tests 44 Chapter 2: Have a Good Command 49 Introduction 50 Concatenating with cat 50 Recording and playback of terminal sessions 53 Finding files and file listing 55 Playing with xargs 63 Translating with tr 69 Checksum and verification 72 Sorting, unique and duplicates 75 Temporary file naming and random numbers 80 Splitting files and data 81 ii Table of Contents Slicing file names based on extension 84 Renaming and moving files in bulk 86 Spell checking and dictionary manipulation 89 Automating interactive input 90 Chapter 3: File In, File Out 95 Introduction 96 Generating files of any size 96 Intersection and set difference (A-B) on text files 97 Finding and deleting duplicate files 100 Making directories for a long path 103 File permissions, ownership, and sticky bit 104 Making files immutable 109 Generating blank files in bulk 110 Finding a symbolic link and its target 111 Enumerating file type statistics 113 Loopback files and mounting 115 Creating ISO files, Hybrid ISO 117 Finding difference between files, patching 120 head and tail – printing the last or first 10 lines 122 Listing only directories – alternative methods 125 Fast command-line navigation using pushd and popd 126 Counting number of lines, words, and characters in a file 128 Printing directory tree 129 Chapter 4: Texting and Driving 131 Introduction 132 Basic regular expression primer 132 Searching and mining "text" inside a file with grep 136 Column-wise cutting of a file with cut 142 Frequency of words used in a given file 146 Basic sed primer 147 Basic awk primer 150 Replacing strings from a text or file 156 Compressing or decompressing JavaScript 158 Iterating through lines, words, and characters in a file 161 Merging multiple files as columns 162 Printing the nth word or column in a file or line 163 Printing text between line numbers or patterns 164 Checking palindrome strings with a script 165 Printing lines in the reverse order 169 Parsing e-mail addresses and URLs from text 171 iii Table of Contents Printing n lines before or after a pattern in a file 172 Removing a sentence in a file containing a word 174 Implementing head, tail, and tac with awk 175 Text slicing and parameter operations 177 Chapter 5: Tangled Web? Not At All! 179 Introduction 180 Downloading from a web page 180 Downloading a web page as formatted plain text 183 A primer on cURL 183 Accessing Gmail from the command line 188 Parsing data from a website 189 Image crawler and downloader 191 Web photo album generator 193 Twitter command-line client 195 define utility with Web backend 197 Finding broken links in a website 199 Tracking changes to a website 200 Posting to a web page and reading response 203 Chapter 6: The Backup Plan 205 Introduction 205 Archiving with tar 206 Archiving with cpio 211 Compressing with gunzip (gzip) 212 Compressing with bunzip (bzip) 215 Compressing with lzma 217 Archiving and compressing with zip 219 squashfs – the heavy compression filesystem 220 Cryptographic tools and hashes 222 Backup snapshots with rsync 224 Version control based backup with Git 227 Cloning hard drive and disks with dd 230 Chapter 7: The Old-boy Network 233 Introduction 233 Basic networking primer 234 Let's ping! 241 Listing all the machines alive on a network 243 Transferring files 247 Setting up an Ethernet and wireless LAN with script 250 Password-less auto-login with SSH 253 Running commands on remote host with SSH 255 29 Getting, setting dates, and delays 30 Debugging the script 33 Functions and arguments 35 Reading the output of a sequence of commands 38 Reading "n" characters without pressing Return 40 Field separators and iterators 41 Comparisons and tests 44 Chapter 2: Have a Good Command 49 Introduction 50 Concatenating with cat 50 Recording and playback of terminal sessions 53 Finding files and file listing 55 Playing with xargs 63 Translating with tr 69 Checksum and verification 72 Sorting, unique and duplicates 75 Temporary file naming and random numbers 80 Splitting files and data 81 ii Table of Contents Slicing file names based on extension 84 Renaming and moving files in bulk 86 Spell checking and dictionary manipulation 89 Automating interactive input 90 Chapter 3: File In, File Out 95 Introduction 96 Generating files of any size 96 Intersection and set difference (A-B) on text files 97 Finding and deleting duplicate files 100 Making directories for a long path 103 File permissions, ownership, and sticky bit 104 Making files immutable 109 Generating blank files in bulk 110 Finding a symbolic link and its target 111 Enumerating file type statistics 113 Loopback files and mounting 115 Creating ISO files, Hybrid ISO 117 Finding difference between files, patching 120 head and tail – printing the last or first 10 lines 122 Listing only directories – alternative methods 125 Fast command-line navigation using pushd and popd 126 Counting number of lines, words, and characters in a file 128 Printing directory tree 129 Chapter 4: Texting and Driving 131 Introduction 132 Basic regular expression primer 132 Searching and mining "text" inside a file with grep 136 Column-wise cutting of a file with cut 142 Frequency of words used in a given file 146 Basic sed primer 147 Basic awk primer 150 Replacing strings from a text or file 156 Compressing or decompressing JavaScript 158 Iterating through lines, words, and characters in a file 161 Merging multiple files as columns 162 Printing the nth word or column in a file or line 163 Printing text between line numbers or patterns 164 Checking palindrome strings with a script 165 Printing lines in the reverse order 169 Parsing e-mail addresses and URLs from text 171 iii Table of Contents Printing n lines before or after a pattern in a file 172 Removing a sentence in a file containing a word 174 Implementing head, tail, and tac with awk 175 Text slicing and parameter operations 177 Chapter 5: Tangled Web? Not At All! 179 Introduction 180 Downloading from a web page 180 Downloading a web page as formatted plain text 183 A primer on cURL 183 Accessing Gmail from the command line 188 Parsing data from a website 189 Image crawler and downloader 191 Web photo album generator 193 Twitter command-line client 195 define utility with Web backend 197 Finding broken links in a website 199 Tracking changes to a website 200 Posting to a web page and reading response 203 Chapter 6: The Backup Plan 205 Introduction 205 Archiving with tar 206 Archiving with cpio 211 Compressing with gunzip (gzip) 212 Compressing with bunzip (bzip) 215 Compressing with lzma 217 Archiving and compressing with zip 219 squashfs – the heavy compression filesystem 220 Cryptographic tools and hashes 222 Backup snapshots with rsync 224 Version control based backup with Git 227 Cloning hard drive and disks with dd 230 Chapter 7: The Old-boy Network 233 Introduction 233 Basic networking primer 234 Let's ping! 241 Listing all the machines alive on a network 243 Transferring files 247 Setting up an Ethernet and wireless LAN with script 250 Password-less auto-login with SSH 253 Running commands on remote host with SSH 255