Steve Mansour sman@scruznet.com Revised: June 5, 1999 (copied by jm /at/ jmason.org from http://www.scruz.net/%7esman/regexp.htm, after the original disappeared! ) What Are Regular Expressions? A regular expression is a formula for matching strings that follow some pattern. Many people are afraid to use them because they can look confusing and complicated. Unfortunately, nothing in this write up can change that. However, I have found that with a bit of practice, it's pretty easy to write these complicated expressions. Plus, once you get the hang of them, you can reduce hours of laborious and error-prone text editing down to minutes or seconds. Regular expressions are supported by many text editors, class libraries such as Rogue Wave's Tools.h++, scripting tools such as awk, grep, sed, and increasingly in interactive development environments such as Microsoft's Visual C++. Regular expressions usage is explained by examples in the sections that follow. Most examples are presented as vi substitution commands or as grep file search commands, but they are representative examples and the concepts can be applied in the use of tools such as sed, awk, perl and other programs that support regular expressions. Have a look at Regular Expressions In Various Tools for examples of regular expression usage in other tools. A short explanation of vi's substitution command and syntax is provided at the end of this document. Regular Expression Basics Regular expressions are made up of normal characters and metacharacters. Normal characters include upper and lower case letters and digits. The metacharacters have special meanings and are described in detail below. In the simplest case, a regular expression looks like a standard search string. For example, the regular expression "testing" contains no metacharacters. It will match "testing" and "123testing" but it will not match "Testing". To really make good use of regular expressions it is critical to understand metacharacters. The table below lists metacharacters and a short explanation of their meaning. Metacharacter Description ------------- ------------------------------------------------------------ . Matches any single character. For example the regular expression r.t would match the strings rat, rut, r t, but not root. $ Matches the end of a line. For example, the regular expression weasel$ would match the end of the string "He's a weasel" but not the string "They are a bunch of weasels." ^ Matches the beginning of a line. For example, the regular expression ^When in would match the beginning of the string "When in the course of human events" but would not match "What and When in the" . * Matches zero or more occurences of the character immediately preceding. For example, the regular expression .* means match any number of any characters. \ This is the quoting character, use it to treat the following character as an ordinary character. For example, \$ is used to match the dollar sign character ($) rather than the end of a line. Similarly, the expression \. is used to match the period character rather than any single character. [ ] Matches any one of the characters between the brackets. For [c1-c2] example, the regular expression r[aou]t matches rat, rot, [^c1-c2] and rut, but not ret. Ranges of characters can specified by using a hyphen. For example, the regular expression [0-9] means match any digit. Multiple ranges can be specified as well. The regular expression [A-Za-z] means match any upper or lower case letter. To match any character except those in the range, the complement range, use the caret as the first character after the opening bracket. For example, the expression [^269A-Z] will match any characters except 2, 6, 9, and upper case letters. \< \> Matches the beginning (\<) or end (\>) or a word. For example, \Replace, then be sure to check the checkbox labled "Regular expression". For vi expressions of the form :%s/pat1/pat2/g set the Find What field to pat1 and the Replace with field to pat2. To simulate the range (% in this case) and the g option you will have to use the Replace All button or appropriate combinations of Find Next and Replace sed Sed is a Stream EDitor which can be used to make changes to files or pipes. For complete details, see the man page sed(1). Here are a few interesting sed scripts. Assume that we're processing a file called price.txt. Note that the edits don't actually happen to the input file, sed simply processes each line of the file with the command you supply and echos the result to its standard out. sed script Description ---------------------------- -------------------------------------------- sed 's/^$/d' price.txt removes all empty lines sed 's/^[ \t]*$/d' price.txt removes all lines containing only whitespace sed 's/"//g' price.txt remove all quotation marks awk Awk is a programming language which can be used to perform sophisticated analysis and manipulation of text data. For complete details, see the man page awk(1). Its peculiar name is an acronym made up of the first character of its authors last names (Aho, Weinberger, and Kernighan). There are many good awk examples in the book The AWK Programming Language (written by Aho, Weinberger, and Kernighan). Please don't form any broad opinions about awk's capabilities based on the following trivial sample scripts. For purposes of these examples, assume that we're working with a file called price.txt. As with sed, awk simply echos its output to its standard out. awk script Description --------------------------------------------------- ---------------------- awk '$0 !~ /^$/' price.txt removes all empty lines a better way to awk 'NF > 0' price.txt remove all lines in awk awk '$2 ~ /^[JT]/ {print $3}' price.txt print the third field of all lines whose second field begins with 'J' or 'T' awk '$2 !~ /[Mm]isc/ {print $3 + $4}' price.txt for all lines whose second field does not contain 'Misc' or 'misc' print the sum of columns 3 and 4 (assumed to be numbers). awk '$3 !~ /^[0-9]+\.[0-9]*$/ {print $0}' price.txt print all lines where field 3 is not a number. The number must be of the form: d.d or d. where d is any number of digits from 0 to 9. awk '$2 ~ /John|Fred/ {print $0}' price.txt print the entire line if the second field contains 'John' or 'Fred' grep grep is a program used to match regular expressions in one or more specified files or in an input stream. Its name programming language which can be used to perform data manipulation on files or pipes. For complete details, see the man page grep(1). Its peculiar name stems from its roots as a command in vi, g/re/p meaning global regular expression print. For the examples below, assume we have the text below in a file named phone.txt. Its format is last name followed by a comma, first name followed by a tab, then a phone number. Francis, John 5-3871 Wong, Fred 4-4123 Jones, Thomas 1-4122 Salazar, Richard 5-2522 grep command Description ----------------------------- -------------------------------------------- grep '\t5-...1' phone.txt print all the lines in phone.txt where the phone number begins with 5 and ends with 1. Note that the tab character is represented by \t. grep '^S[^ ]* R' phone.txt print lines where the last name begins with S and first name begins with R. grep '^[JW]' phone.txt print lines where the last name begins with J or W grep ', ....\t' phone.txt print lines where the first name is 4 characters. The tab character is represented by \t. grep -v '^[JW]' phone.txt print lines that do not begin with J or W grep '^[M-Z]' phone.txt print lines where the last name begins with any letter from M to Z. grep '^[M-Z].*[12]' phone.txt print lines where the last name begins with a letter from M to Z and where the phone number ends with a 1 or 2. egrep egrep is an extended version of grep. It supports a few more metacharacters in its regular expressions. For the examples below, assume we have the text below in a file named phone.txt. Its format is last name followed by a comma, first name followed by a tab, then a phone number. Francis, John 5-3871 Wong, Fred 4-4123 Jones, Thomas 1-4122 Salazar, Richard 5-2522 egrep command Description ------------------------------ ------------------------------------------- egrep '(John|Fred)' phone.txt print all lines that contain the name John or Fred. egrep 'John|22$|^W' phone.txt print lines that contain John or that end with 22 or that begin with W. egrep 'net(work)?s' report.txt print lines in report.txt contain networks or nets.