Learn/Bash/8. Text processing and a small project
Lesson 8 of 8
Sign in

Bash, lesson 8

Text processing and a small project

Bash scripts spend much of their time handling text: file names, log lines, settings. You can do a surprising amount without calling any external program, using parameter expansion, the ${...} syntax you already met for arrays.

Length and slices

Positions start at 0. The space in ${word: -3} matters: without it, Bash reads :- as a different operator (a default value, below).

Changing case

Replacing

${var/old/new} replaces the first match, and ${var//old/new} replaces every match. The old part is a pattern, so * and ? work as in :

2 exercises to try what you just read.

Exercise 1

Tidy file names

Press Ctrl Enter or Check to test your code.

Exercise 2

Log report

Press Ctrl Enter or Check to test your code.

case

Trimming from the ends

Four operators cut a pattern from the start or end of a value. A single sign removes the shortest match, a double sign the longest:

ExpansionRemovesf=report.tar.gz gives
${f#*.}shortest match from the starttar.gz
${f##*.}longest match from the startgz (the extension)
${f%.*}shortest match from the endreport.tar
${f%%.*}longest match from the endreport

A memory aid: on a keyboard, # is left of $ and % is right of it, so # trims the left and % the right.

Defaults

${var:-fallback} gives fallback when var is unset or empty, without changing var. It is an easy way to handle missing input: read -r name; echo "Hi, ${name:-stranger}".

Splitting on other characters

read splits on whitespace because of the special variable IFS (internal field separator). Set it just for one read to split on something else, such as commas:

Writing IFS=, directly before read changes it only for that command. An empty IFS= turns splitting off completely, so IFS= read -r line keeps the line exactly as it was, including spaces at the start and end. <<< (a "here string") feeds a string to a command's standard input, which is handy for testing.

A small project

The second exercise combines the course: reading lines in a loop, splitting them, a case, counters in an associative array, a function, arithmetic and formatted output with printf. Write it step by step, running after each change: first echo each parsed line, then count, then format the report.

Further reading

  • Bash manual: Shell parameter expansion
  • Bash manual: Word splitting and IFS
  • Bash manual: Here strings

Each input line is a file name. Make a tidy version: all lower case, with every space replaced by an underscore. Print old -> new for each line.

Sample input:

Sample output:

Each input line is a log entry: a time, a level (INFO, WARN or ERROR) and a message, for example 09:14 ERROR disk full. Print a report:

  1. One line per level, always in the order INFO, WARN, ERROR, formatted with printf "%-6s%3d\n" LEVEL COUNT.
  2. First error: TIME MESSAGE for the first ERROR line, or First error: none if there was none.
  3. Other lines: N, the number of lines whose level was not one of the three.

Write a function print_count that takes a level and prints its line of the report.

Sample input:

Sample output:

word="Bashful"
echo "${#word}"        # 7, the number of characters
echo "${word:0:4}"     # Bash: 4 characters from position 0
echo "${word:4}"       # ful: from position 4 to the end
echo "${word: -3}"     # ful: the last 3 (note the space before -)
name="ada Lovelace"
echo "${name^}"        # Ada Lovelace: first character upper case
echo "${name^^}"       # ADA LOVELACE: everything upper case
echo "${name,,}"       # ada lovelace: everything lower case
path="my holiday photo.jpg"
echo "${path/ /_}"     # my_holiday photo.jpg
echo "${path// /_}"    # my_holiday_photo.jpg
IFS=, read -r item qty price <<< "pen,3,120"
echo "$item costs $price"      # pen costs 120
Holiday Photo.JPG
My Song Final.MP3
notes.txt
Holiday Photo.JPG -> holiday_photo.jpg
My Song Final.MP3 -> my_song_final.mp3
notes.txt -> notes.txt
09:00 INFO service started
09:14 ERROR disk full
09:15 WARN retrying
09:20 ERROR disk still full
09:30 DEBUG cache stats
INFO    1
WARN    1
ERROR   2
First error: 09:14 disk full
Other lines: 1