Introduction to string processing
=================================

stringproc.lisp enlarges Maximas capabilities of working with strings and adds some useful functions for file in/output.

For questions and bugs please mail to van.nek at arcor.de .

In Maxima a string is easily constructed by typing "text". Note that Maxima-strings are no Lisp-strings and vice versa. stringp tests for Maxima strings, and lstringp tests for Lisp strings. If for some reasons you have a value, that is a Lisp-string, maybe when using Maxima-function sconcat, you can convert via sunlisp.

     (%i1) m: "text";
     (%o1)                         text
     (%i2) [stringp(m),lstringp(m)];
     (%o2)                     [true, false]
     (%i3) l: sconcat("text");
     (%o3)                         text
     (%i4) [stringp(l),lstringp(l)];
     (%o4)                     [false, true]
     (%i5) stringp( sunlisp(l) );
     (%o5)                         true

All functions in stringproc.lisp, that return strings, return Maxima-strings.

Characters are introduced as Maxima-strings of length 1. Of course, these are no Lisp-characters. Tests can be done with charp (respectively lcharp and conversion from Lisp to Maxima with cunlisp).

     (%i1) c: "e";
     (%o1)                           e
     (%i2) [charp(c),lcharp(c)];
     (%o2)                     [true, false]
     (%i3) supcase(c);
     (%o3)                           E
     (%i4) charp(%);
     (%o4)                         true

Again, all functions in stringproc.lisp, that return characters, return Maxima-characters. Due to the fact, that the introduced characters are strings of length 1, you can use a lot of string functions also for characters. As seen, supcase is one example.

It is important to know, that the first character in a Maxima-string is at position 1. This is designed due to the fact that the first element in a Maxima-list is at position 1 too. See definitions of charat and charlist for examples.

In applications string-functions are often used when working with files. You will find some useful stream- and print-functions in stringproc.lisp. The following example shows some of the here introduced functions at work.

Example:

openw returns an output stream to a file, printf then allows formatted writing to this file. See printf for details.

     (%i1) s: openw("E:/file.txt");
     (%o1)                    #<output stream E:/file.txt>
     (%i2) for n:0 thru 10 do printf( s, "~d ", fib(n) );
     (%o2)                                done
     (%i3) printf( s, "~%~d ~f ~a ~a ~f ~e ~a~%",
                   42,1.234,sqrt(2),%pi,1.0e-2,1.0e-2,1.0b-2 );
     (%o3)                                false
     (%i4) close(s);
     (%o4)                                true

After closing the stream you can open it again, this time with input direction. readline returns the entire line as one string. The stringproc package now offers a lot of functions for manipulating strings. Tokenizing can be done by split or tokens.

     (%i5) s: openr("E:/file.txt");
     (%o5)                     #<input stream E:/file.txt>
     (%i6) readline(s);
     (%o6)                     0 1 1 2 3 5 8 13 21 34 55
     (%i7) line: readline(s);
     (%o7)               42 1.234 sqrt(2) %pi 0.01 1.0E-2 1.0b-2
     (%i8) list: tokens(line);
     (%o8)           [42, 1.234, sqrt(2), %pi, 0.01, 1.0E-2, 1.0b-2]
     (%i9) map( parsetoken, list );
     (%o9)           [42, 1.234, false, false, 0.01, 0.01, false]

parsetoken only parses integer and float numbers. Parsing symbols or bigfloats needs parse_string, which will be autoloaded from eval_string.lisp.

     (%i10) map( parse_string, list );
     (%o10)           [42, 1.234, sqrt(2), %pi, 0.01, 0.01, 1.0b-2]
     (%i11) float(%);
     (%o11) [42.0, 1.234, 1.414213562373095, 3.141592653589793, 0.01,
                                                          0.01, 0.01]
     (%i12) readline(s);
     (%o12)                               false
     (%i13) close(s)$

readline returns false when the end of file occurs.

Functions and Variables for input and output
============================================

Example:

     (%i1) s: openw("E:/file.txt");
     (%o1)                     #<output stream E:/file.txt>
     (%i2) control:
     "~2tAn atom: ~20t~a~%~2tand a list: ~20t~{~r ~}~%~2t\
     and an integer: ~20t~d~%"$
     (%i3) printf( s,control, 'true,[1,2,3],42 )$
     (%o3)                                false
     (%i4) close(s);
     (%o4)                                true
     (%i5) s: openr("E:/file.txt");
     (%o5)                     #<input stream E:/file.txt>
     (%i6) while stringp( tmp:readline(s) ) do print(tmp)$
       An atom:          true
       and a list:       one two three
       and an integer:   42
     (%i7) close(s)$

 -- Function: close (STREAM)
     Closes STREAM and returns true if STREAM had been open.

 -- Function: flength (STREAM)
     Returns the number of elements in STREAM.

 -- Function: fposition (STREAM)
 -- Function: fposition (STREAM, POS)
     Returns the current position in STREAM, if POS is not used. If POS is used, fposition sets the position in STREAM. POS has to be a positive number, the first element in STREAM is in position 1.

 -- Function: freshline ()
 -- Function: freshline (STREAM)
     Writes a new line (to STREAM), if the position is not at the beginning of a line. See also newline.

 -- Function: newline ()
 -- Function: newline (STREAM)
     Writes a new line (to STREAM). See sprint for an example of using newline(). Note that there are some cases, where newline() does not work as expected.

 -- Function: opena (FILE)
     Returns an output stream to FILE. If an existing file is opened, opena appends elements at the end of file.

 -- Function: openr (FILE)
     Returns an input stream to FILE. If FILE does not exist, it will be created.

 -- Function: openw (FILE)
     Returns an output stream to FILE. If FILE does not exist, it will be created. If an existing file is opened, openw destructively modifies FILE.

 -- Function: printf (DEST, STRING)
 -- Function: printf (DEST, STRING, EXPR_1, ..., EXPR_N)
     Makes the Common Lisp function FORMAT available in Maxima. (From gcl.info: "format produces formatted output by outputting the characters of control-string string and observing that a tilde introduces a directive. The character after the tilde, possibly preceded by prefix parameters and modifiers, specifies what kind of formatting is desired. Most directives use one or more elements of args to create their output.")

     The following description and the examples may give an idea of using printf. See a Lisp reference for more information.

             ~%       new line
             ~&       fresh line
             ~t       tab
             ~$       monetary
             ~d       decimal integer
             ~b       binary integer
             ~o       octal integer
             ~x       hexadecimal integer
             ~br      base-b integer
             ~r       spell an integer
             ~p       plural
             ~f       floating point
             ~e       scientific notation
             ~g       ~f or ~e, depending upon magnitude
             ~a       as printed by Maxima function print
             ~s       strings enclosed in "double quotes"
             ~~       ~
             ~<       justification, ~> terminates
             ~(       case conversion, ~) terminates
             ~[       selection, ~] terminates
             ~{       iteration, ~} terminates

     Please note that there is no format specifier for bigfloats. However bigfloats can simply be printed by using the ~a directive. ~s prints strings enclosed in "double quotes", you can avoid this by using ~a. Note that the selection directive ~[ is zero-indexed. Also note that there are some directives, which do not work in Maxima. For example, ~:[ fails.

          (%i1) printf( false, "~a ~a ~4f ~a ~@r",
                        "String",sym,bound,sqrt(12),144), bound = 1.234;
          (%o1)                 String sym 1.23 2*sqrt(3) CXLIV
          (%i2) printf( false,"~{~a ~}",["one",2,"THREE"] );
          (%o2)                          one 2 THREE
          (%i3) printf(true,"~{~{~9,1f ~}~%~}",mat ),
                    mat = args(matrix([1.1,2,3.33],[4,5,6],[7,8.88,9]))$
                1.1       2.0       3.3
                4.0       5.0       6.0
                7.0       8.9       9.0
          (%i4) control: "~:(~r~) bird~p ~[is~;are~] singing."$
          (%i5) printf( false,control, n,n,if n=1 then 0 else 1 ), n=2;
          (%o5)                    Two birds are singing.

     If DEST is a stream or true, then printf returns false. Otherwise, printf returns a string containing the output.

 -- Function: readline (STREAM)
     Returns a string containing the characters from the current position in STREAM up to the end of the line or FALSE if the end of the file is encountered.

 -- Function: sprint (EXPR_1, ..., EXPR_N)
     Evaluates and displays its arguments one after the other `on a line' starting at the leftmost position. The numbers are printed with the '-' right next to the number, and it disregards line length. newline(), which will be autoloaded from stringproc.lisp might be useful, if you whish to place intermediate line breaking.

          (%i1) for n:0 thru 20 do sprint( fib(n) )$
          0 1 1 2 3 5 8 13 21 34 55 89 144 233 377 610 987 1597 2584 4181 6765
          (%i2) for n:0 thru 22 do (
                   sprint(fib(n)), if mod(n,10)=9 then newline() )$
          0 1 1 2 3 5 8 13 21 34
          55 89 144 233 377 610 987 1597 2584 4181
          6765 10946 17711

Functions and Variables for characters
======================================

 -- Function: alphacharp (CHAR)
     Returns true if CHAR is an alphabetic character.

 -- Function: alphanumericp (CHAR)
     Returns true if CHAR is an alphabetic character or a digit.

 -- Function: ascii (INT)
     Returns the character corresponding to the ASCII number INT. ( -1 < int < 256 )

          (%i1) for n from 0 thru 255 do (
             tmp: ascii(n), if alphacharp(tmp) then sprint(tmp),
                if n=96 then newline() )$
          A B C D E F G H I J K L M N O P Q R S T U V W X Y Z
          a b c d e f g h i j k l m n o p q r s t u v w x y z

 -- Function: cequal (CHAR_1, CHAR_2)
     Returns true if CHAR_1 and CHAR_2 are the same.

 -- Function: cequalignore (CHAR_1, CHAR_2)
     Like cequal but ignores case.

 -- Function: cgreaterp (CHAR_1, CHAR_2)
     Returns true if the ASCII number of CHAR_1 is greater than the number of CHAR_2.

 -- Function: cgreaterpignore (CHAR_1, CHAR_2)
     Like cgreaterp but ignores case.

 -- Function: charp (OBJ)
     Returns true if OBJ is a Maxima-character. See introduction for example.

 -- Function: cint (CHAR)
     Returns the ASCII number of CHAR.

 -- Function: clessp (CHAR_1, CHAR_2)
     Returns true if the ASCII number of CHAR_1 is less than the number of CHAR_2.

 -- Function: clesspignore (CHAR_1, CHAR_2)
     Like clessp but ignores case.

 -- Function: constituent (CHAR)
     Returns true if CHAR is a graphic character and not the space character. A graphic character is a character one can see, plus the space character. (constituent is defined by Paul Graham, ANSI Common Lisp, 1996, page 67.)

          (%i1) for n from 0 thru 255 do (
          tmp: ascii(n), if constituent(tmp) then sprint(tmp) )$
          ! " #  %  ' ( ) * + , - . / 0 1 2 3 4 5 6 7 8 9 : ; < = > ? @ A B
          C D E F G H I J K L M N O P Q R S T U V W X Y Z [ \ ] ^ _ ` a b c
          d e f g h i j k l m n o p q r s t u v w x y z { | } ~

 -- Function: cunlisp (LISP_CHAR)
     Converts a Lisp-character into a Maxima-character. (You won't need it.)

 -- Function: digitcharp (CHAR)
     Returns true if CHAR is a digit.

 -- Function: lcharp (OBJ)
     Returns true if OBJ is a Lisp-character. (You won't need it.)

 -- Function: lowercasep (CHAR)
     Returns true if CHAR is a lowercase character.

 -- Variable: newline
     The newline character.

 -- Variable: space
     The space character.

 -- Variable: tab
     The tab character.

 -- Function: uppercasep (CHAR)
     Returns true if CHAR is an uppercase character.

Functions and Variables for strings
===================================

 -- Function: sunlisp (LISP_STRING)
     Converts a Lisp-string into a Maxima-string. (In general you won't need it.)

 -- Function: lstringp (OBJ)
     Returns true if OBJ is a Lisp-string. (In general you won't need it.)

 -- Function: stringp (OBJ)
     Returns true if OBJ is a Maxima-string. See introduction for example.

 -- Function: charat (STRING, N)
     Returns the N-th character of STRING. The first character in STRING is returned with N = 1.

          (%i1) charat("Lisp",1);
          (%o1)                           L

 -- Function: charlist (STRING)
     Returns the list of all characters in STRING.

          (%i1) charlist("Lisp");
          (%o1)                     [L, i, s, p]
          (%i2) %[1];
          (%o2)                           L

 -- Function: parsetoken (STRING)
     parsetoken converts the first token in STRING to the corresponding number or returns false if the number cannot be determined . The delimiter set for tokenizing is {space, comma, semicolon, tab, newline}.

          (%i1) 2*parsetoken("1.234 5.678");
          (%o1)                         2.468

     For parsing you can also use function parse_string. See description in file 'share\contrib\eval_string.lisp'.

 -- Function: sconc (EXPR_1, ..., EXPR_N)
     Evaluates its arguments and concatenates them into a string. sconc is like sconcat but returns a Maxima string.

          (%i1) sconc("xx[",3,"]:",expand((x+y)^3));
          (%o1)             xx[3]:y^3+3*x*y^2+3*x^2*y+x^3
          (%i2) stringp(%);
          (%o2)                         true

 -- Function: scopy (STRING)
     Returns a copy of STRING as a new string.

 -- Function: sdowncase (STRING)
 -- Function: sdowncase (STRING, START)
 -- Function: sdowncase (STRING, START, END)
     Like supcase, but uppercase characters are converted to lowercase.

 -- Function: sequal (STRING_1, STRING_2)
     Returns true if STRING_1 and STRING_2 are the same length and contain the same characters.

 -- Function: sequalignore (STRING_1, STRING_2)
     Like sequal but ignores case.

 -- Function: sexplode (STRING)
     sexplode is an alias for function charlist.

 -- Function: simplode (LIST)
 -- Function: simplode (LIST, DELIM)
     simplode takes a list of expressions and concatenates them into a string. If no delimiter DELIM is used, simplode is like sconc and uses no delimiter. DELIM can be any string.

          (%i1) simplode(["xx[",3,"]:",expand((x+y)^3)]);
          (%o1)             xx[3]:y^3+3*x*y^2+3*x^2*y+x^3
          (%i2) simplode( sexplode("stars")," * " );
          (%o2)                   s * t * a * r * s
          (%i3) simplode( ["One","more","coffee."]," " );
          (%o3)                   One more coffee.

 -- Function: sinsert (SEQ, STRING, POS)
     Returns a string that is a concatenation of substring (STRING, 1, POS - 1), the string SEQ and substring (STRING, POS). Note that the first character in STRING is in position 1.

          (%i1) s: "A submarine."$
          (%i2) sconc( substring(s,1,3),"yellow ",substring(s,3) );
          (%o2)                  A yellow submarine.
          (%i3) sinsert("hollow ",s,3);
          (%o3)                  A hollow submarine.

 -- Function: sinvertcase (STRING)
 -- Function: sinvertcase (STRING, START)
 -- Function: sinvertcase (STRING, START, END)
     Returns STRING except that each character from position START to END is inverted. If END is not given, all characters from start to the END of STRING are replaced.

          (%i1) sinvertcase("sInvertCase");
          (%o1)                      SiNVERTcASE

 -- Function: slength (STRING)
     Returns the number of characters in STRING.

 -- Function: smake (NUM, CHAR)
     Returns a new string with a number of NUM characters CHAR.

          (%i1) smake(3,"w");
          (%o1)                          www

 -- Function: smismatch (STRING_1, STRING_2)
 -- Function: smismatch (STRING_1, STRING_2, TEST)
     Returns the position of the first character of STRING_1 at which STRING_1 and STRING_2 differ or false. Default test function for matching is sequal. If smismatch should ignore case, use sequalignore as test.

          (%i1) smismatch("seven","seventh");
          (%o1)                           6

 -- Function: split (STRING)
 -- Function: split (STRING, DELIM)
 -- Function: split (STRING, DELIM, MULTIPLE)
     Returns the list of all tokens in STRING. Each token is an unparsed string. split uses DELIM as delimiter. If DELIM is not given, the space character is the default delimiter. MULTIPLE is a boolean variable with true by default. Multiple delimiters are read as one. This is useful if tabs are saved as multiple space characters. If MULTIPLE is set to false, each delimiter is noted.

          (%i1) split("1.2   2.3   3.4   4.5");
          (%o1)                 [1.2, 2.3, 3.4, 4.5]
          (%i2) split("first;;third;fourth",";",false);
          (%o2)               [first, , third, fourth]

 -- Function: sposition (CHAR, STRING)
     Returns the position of the first character in STRING which matches CHAR. The first character in STRING is in position 1. For matching characters ignoring case see ssearch.

 -- Function: sremove (SEQ, STRING)
 -- Function: sremove (SEQ, STRING, TEST)
 -- Function: sremove (SEQ, STRING, TEST, START)
 -- Function: sremove (SEQ, STRING, TEST, START, END)
     Returns a string like STRING but without all substrings matching SEQ. Default test function for matching is sequal. If sremove should ignore case while searching for SEQ, use sequalignore as test. Use START and END to limit searching. Note that the first character in STRING is in position 1.

          (%i1) sremove("n't","I don't like coffee.");
          (%o1)                   I do like coffee.
          (%i2) sremove ("DO ",%,'sequalignore);
          (%o2)                    I like coffee.

 -- Function: sremovefirst (SEQ, STRING)
 -- Function: sremovefirst (SEQ, STRING, TEST)
 -- Function: sremovefirst (SEQ, STRING, TEST, START)
 -- Function: sremovefirst (SEQ, STRING, TEST, START, END)
     Like sremove except that only the first substring that matches seq is removed.

 -- Function: sreverse (STRING)
     Returns a string with all the characters of STRING in reverse order.

 -- Function: ssearch (SEQ, STRING)
 -- Function: ssearch (SEQ, STRING, TEST)
 -- Function: ssearch (SEQ, STRING, TEST, START)
 -- Function: ssearch (SEQ, STRING, TEST, START, END)
     Returns the position of the first substring of STRING that matches the string SEQ. Default test function for matching is sequal. If ssearch should ignore case, use sequalignore as test. Use START and END to limit searching. Note that the first character in STRING is in position 1.

          (%i1) ssearch("~s","~{~S ~}~%",'sequalignore);
          (%o1)                                  4

 -- Function: ssort (STRING)
 -- Function: ssort (STRING, TEST)
     Returns a string that contains all characters from STRING in an order such there are no two successive characters C and D such that test (C, D) is false and test (D, C) is true. Default test function for sorting is CLESSP. The set of test functions is {clessp, clesspignore, cgreaterp, cgreaterpignore, cequal, cequalignore}.

          (%i1) ssort("I don't like Mondays.");
          (%o1)                    '.IMaddeiklnnoosty
          (%i2) ssort("I don't like Mondays.",'cgreaterpignore);
          (%o2)                 ytsoonnMlkIiedda.'

 -- Function: ssubst (NEW, OLD, STRING)
 -- Function: ssubst (NEW, OLD, STRING, TEST)
 -- Function: ssubst (NEW, OLD, STRING, TEST, START)
 -- Function: ssubst (NEW, OLD, STRING, TEST, START, END)
     Returns a string like STRING except that all substrings matching OLD are replaced by NEW. OLD and NEW need not to be of the same length. Default test function for matching is sequal. If ssubst should ignore case while searching for old, use sequalignore as test. Use START and END to limit searching. Note that the first character in STRING is in position 1.

          (%i1) ssubst("like","hate","I hate Thai food. I hate green tea.");
          (%o1)          I like Thai food. I like green tea.
          (%i2) ssubst("Indian","thai",%,'sequalignore,8,12);
          (%o2)         I like Indian food. I like green tea.

 -- Function: ssubstfirst (NEW, OLD, STRING)
 -- Function: ssubstfirst (NEW, OLD, STRING, TEST)
 -- Function: ssubstfirst (NEW, OLD, STRING, TEST, START)
 -- Function: ssubstfirst (NEW, OLD, STRING, TEST, START, END)
     Like subst except that only the first substring that matches OLD is replaced.

 -- Function: strim (SEQ,STRING)
     Returns a string like STRING, but with all characters that appear in SEQ removed from both ends.

          (%i1) "/* comment */"$
          (%i2) strim(" /*",%);
          (%o2)                        comment
          (%i3) slength(%);
          (%o3)                           7

 -- Function: striml (SEQ, STRING)
     Like strim except that only the left end of STRING is trimmed.

 -- Function: strimr (SEQ, STRING)
     Like strim except that only the right end of string is trimmed.

 -- Function: substring (STRING, START)
 -- Function: substring (STRING, START, END)
     Returns the substring of STRING beginning at position START and ending at position END. The character at position END is not included. If END is not given, the substring contains the rest of the string. Note that the first character in STRING is in position 1.

          (%i1) substring("substring",4);
          (%o1)                        string
          (%i2) substring(%,4,6);
          (%o2)                          in

 -- Function: supcase (STRING)
 -- Function: supcase (STRING, START)
 -- Function: supcase (STRING, START, END)
     Returns STRING except that lowercase characters from position START to END are replaced by the corresponding uppercase ones. If END is not given, all lowercase characters from START to the end of STRING are replaced.

          (%i1) supcase("english",1,2);
          (%o1)                        English

 -- Function: tokens (STRING)
 -- Function: tokens (STRING, TEST)
     Returns a list of tokens, which have been extracted from STRING. The tokens are substrings whose characters satisfy a certain test function. If test is not given, CONSTITUENT is used as the default test. {constituent, alphacharp, digitcharp, lowercasep, uppercasep, charp, characterp, alphanumericp} is the set of test functions. (The Lisp-version of tokens is written by Paul Graham. ANSI Common Lisp, 1996, page 67.)

          (%i1) tokens("24 October 2005");
          (%o1)                  [24, October, 2005]
          (%i2) tokens("05-10-24",'digitcharp);
          (%o2)                     [05, 10, 24]
          (%i3) map(parsetoken,%);
          (%o3)                      [5, 10, 24]