Lisp closures, arguments, and macros Lee Spector, lspector@hampshire.edu, 1995-1998 Common Lisp provides considerable flexibility in the specification of function parameters. Ordinary function definitions specify some number of REQUIRED parameters; calls to these functions must pass exactly the correct number of arguments: (defun gimme-two (the-first the-second) (list 'here-they-are the-first the-second)) --> GIMME-TWO (gimme-two 1 2) --> (HERE-THEY-ARE 1 2) (gimme-two 1) > Error: Too few arguments. (gimme-two 1 2 3) > Error: Too many arguments. The first enhancement that we'll look at is optional parameters. A function is specified to take optional parameters by putting &optional, followed by parameter names, in the parameter list. (defun shout (stuff &optional more-stuff) (list 'hey 'you! '--- stuff more-stuff '!)) --> SHOUT (shout 'look 'out) --> (HEY YOU! --- LOOK OUT !) (shout 'look) --> (HEY YOU! --- LOOK NIL !) (defun shout (stuff &optional more-stuff) (if more-stuff (list 'hey 'you! '--- stuff more-stuff '!) (list 'hey 'you! '--- stuff '!))) --> SHOUT (shout 'look 'out) --> (HEY YOU! --- LOOK OUT !) (shout 'look) --> (HEY YOU! --- LOOK !) If you don't want your optional parameter to default to nil, you can provide another default value: (defun shout (stuff &optional (more-stuff 'around)) (list 'hey 'you! '--- stuff more-stuff '!)) --> SHOUT (shout 'look 'out) --> (HEY YOU! --- LOOK OUT !) (shout 'look) --> (HEY YOU! --- LOOK AROUND !) You can specify as many optional arguments as you want: (defun fill-grocery-bag (item &optional (item2 'milk) (item3 'eggs)) (list 'the 'bag 'contains item item2 'and item3)) --> FILL-GROCERY-BAG (fill-grocery-bag 'spam) --> (THE BAG CONTAINS SPAM MILK AND EGGS) (fill-grocery-bag 'spam 'flan) --> (THE BAG CONTAINS SPAM FLAN AND EGGS) (fill-grocery-bag 'spam 'flan 'marjoram) --> (THE BAG CONTAINS SPAM FLAN AND MARJORAM) (fill-grocery-bag 'spam 'flan 'marjoram 'dan) > Error: Too many arguments. You can also specify SUPPLIED-P variables with your optional parameters: (defun eat (what &optional (when 'supper when-supplied)) (if (and (eq when 'supper) when-supplied) (print '(supper is the default -- you could have left it out))) (list 'you 'ate what 'at when)) --> EAT (eat 'gefilte-fish 'lunch) --> (YOU ATE GEFILTE-FISH AT LUNCH) (eat 'gefilte-fish) --> (YOU ATE GEFILTE-FISH AT SUPPER) (eat 'gefilte-fish 'supper) (SUPPER IS THE DEFAULT -- YOU COULD HAVE LEFT IT OUT) --> (YOU ATE GEFILTE-FISH AT SUPPER) We can write functions that are even more flexible by using &rest parameters. An &rest parameter gets bound to ALL remaining arguments in the function call. (defun eat (first-food &rest more-foods) (format t "~%you started with ~A" first-food) (dolist (next-food more-foods) (format t "~%and then you had ~A" next-food)) 'burp) --> EAT (eat 'gefilte-fish) you started with GEFILTE-FISH --> BURP (eat 'gefilte-fish 'horseradish) you started with GEFILTE-FISH and then you had HORSERADISH --> BURP (eat 'gefilte-fish 'horseradish 'pickles) you started with GEFILTE-FISH and then you had HORSERADISH and then you had PICKLES --> BURP (eat 'gefilte-fish 'horseradish 'pickles 'olives) you started with GEFILTE-FISH and then you had HORSERADISH and then you had PICKLES and then you had OLIVES --> BURP The &keywords facility allows us to specify NAMED optional parameters. (defun make-poem (title &key (subject 'flower)) (format t "~%~A, by Lulu Lispy" title) (format t "~%--------------------------") (format t "~%'twas a stupendously beautiful ~A" subject) (format t "~%upon my snow-covered porch") (format t "~%such a lovely lovely ~A" subject) (format t "~%I blasted it with my torch.")) --> MAKE-POEM (make-poem 'spring) SPRING, by Lulu Lispy -------------------------- 'twas a stupendously beautiful FLOWER upon my snow-covered porch such a lovely lovely FLOWER I blasted it with my torch. --> NIL (make-poem 'spring :subject 'cat) SPRING, by Lulu Lispy -------------------------- 'twas a stupendously beautiful CAT upon my snow-covered porch such a lovely lovely CAT I blasted it with my torch. --> NIL Note that the keyword must be provided. This causes an error: (make-poem 'spring 'cat) > Error: Incorrect keyword arguments in (CAT) . The nifty thing about keyword parameters is that you can specify any subset of a function's arguments, in any order: (defun make-poem (title &key (subject 'flower) (location 'porch) (blaster 'torch) (covering 'snow)) (format t "~%~A, by Lulu Lispy" title) (format t "~%--------------------------") (format t "~%'twas a stupendously beautiful ~A" subject) (format t "~%upon my ~A-covered ~A" covering location) (format t "~%such a lovely lovely ~A" subject) (format t "~%I blasted it with my ~A." blaster)) --> MAKE-POEM (make-poem 'spring) SPRING, by Lulu Lispy -------------------------- 'twas a stupendously beautiful FLOWER upon my SNOW-covered PORCH such a lovely lovely FLOWER I blasted it with my TORCH. --> NIL (make-poem 'spring :location 'veranda :blaster 'power-sander) SPRING, by Lulu Lispy -------------------------- 'twas a stupendously beautiful FLOWER upon my SNOW-covered VERANDA such a lovely lovely FLOWER I blasted it with my POWER-SANDER. --> NIL (make-poem 'spring :location 'veranda :blaster 'power-sander :subject 'pumpkin) SPRING, by Lulu Lispy -------------------------- 'twas a stupendously beautiful PUMPKIN upon my SNOW-covered VERANDA such a lovely lovely PUMPKIN I blasted it with my POWER-SANDER. --> NIL (make-poem 'spring :subject 'space-alien :location 'veranda :blaster 'power-sander :covering 'glitter) SPRING, by Lulu Lispy -------------------------- 'twas a stupendously beautiful SPACE-ALIEN upon my GLITTER-covered VERANDA such a lovely lovely SPACE-ALIEN I blasted it with my POWER-SANDER. --> NIL &keyword parameters can also be given SUPPLIED-P variables, and they may be combined with &optional and &rest parameters. Note that many built-in functions use &keyword parameters for optional features. For example, MEMBER has 3 keyword parameters: TEST, TEST-NOT, and KEY. (It sounds funny to have a &keyword parameter called KEY, but that's just a coincidence.) Here are some examples: (member 'prune '(raisin prune cheese)) --> (PRUNE CHEESE) (defun young-version (food) (case food (raisin 'grape) (prune 'plum) (cheese 'milk))) --> YOUNG-VERSION (young-version 'prune) --> PLUM (member 'plum '(raisin prune cheese)) --> NIL (member 'plum '(raisin prune cheese) :key #'young-version) --> (PRUNE CHEESE) (member 'green '((blue sea)(green grass) (yellow teeth))) --> NIL (member 'green '((blue sea)(green grass) (yellow teeth)) :key #'car) --> ((GREEN GRASS) (YELLOW TEETH)) NOTE: KEY defaults to the function IDENTITY. (member '(the jetsons) '((the flintstones)(the jetsons) (the honeymooners))) --> NIL (member '(the jetsons) '((the flintstones)(the jetsons) (the honeymooners)) :test #'equalp) --> ((THE JETSONS) (THE HONEYMOONERS)) NOTE: TEST defaults to the function EQ. TEST-NOT is not as obvious, but it is sometimes handy (member 4 '(4 4 4 4 4 4 23 5 5 5 5 5 5)) --> (4 4 4 4 4 4 23 5 5 5 5 5 5) (member 4 '(4 4 4 4 4 4 23 5 5 5 5 5 5) :test-not #'eq) --> (23 5 5 5 5 5 5) Closures A function + its environment is called a "closure" Any function (or lambda form) defined in a non-null lexical environment -- that is, not at the top level -- is actually a closure. Closures can be used in some powerful ways. The following gives us many of the advantages of global variables without the disadvantages. (let ((counter 0)) (defun new-id () (incf counter)) (defun reset-id () (setq counter 0))) --> RESET-ID (new-id) --> 1 (new-id) --> 2 (new-id) --> 3 (reset-id) --> 0 (new-id) --> 1 (new-id) --> 2 We can also return closures from functions and then call them with FUNCALL: (defun make-adder (n) #'(lambda (x) (+ x n))) --> MAKE-ADDER (funcall (make-adder 2) 5) --> 7 (funcall (make-adder 12) 500) --> 512 It is also possible to make closures that can be asked to change their state (defun make-adderb (n) #'(lambda (x &optional change) (if change (setq n x) (+ x n)))) --> MAKE-ADDERB (setq addx (make-adderb 1)) --> # (funcall addx 3) --> 4 (funcall addx 100 t) --> 100 (funcall addx 3) --> 103 Macros We've used some things that look like functions but don't evaluate their arguments. Examples are SETQ and SETF: (setq mylist (alpha beta gamma habbadabba)) --> (ALPHA BETA GAMMA HABBADABBA) (nth 1 mylist) --> BETA (setf (nth 1 mylist) 'omega) --> OMEGA mylist --> (ALPHA OMEGA GAMMA HABBADABBA) What's really going on in the call to setf? Setf is a MACRO. Macros are pieces of code that transform themselves into other code before being evaluated. The arguments are not evaluated. (macroexpand '(setf (nth 1 mylist) 'omega)) --> (%SETNTH 1 MYLIST 'OMEGA) MACROEXPAND form &optional environment [Function] expands form repeatedly within environment until it is no longer a macro call, and returns the expansion and a second value, t if form was a macro call and nil if it was not. Another example: COND (setq x 5) --> 5 (setq y 8) --> 8 (cond ((> x y) 'x-higher) ((< x y) 'x-lower) (t 'same)) --> X-LOWER (macroexpand '(cond ((> x y) 'x-higher) ((< x y) 'x-lower) (t 'same))) --> (IF (> X Y) (PROGN 'X-HIGHER) (COND ((< X Y) 'X-LOWER) (T 'SAME))) Another example: DEFUN (macroexpand '(defun binky (n) (+ n 2))) --> (PROGN (EVAL-WHEN (COMPILE-TOPLEVEL) (NOTE-FUNCTION-INFO 'BINKY '(LAMBDA (N) (DECLARE (GLOBAL-FUNCTION-NAME BINKY)) (BLOCK BINKY (+ N 2))) NIL)) (%DEFUN (NFUNCTION BINKY (LAMBDA (N) (DECLARE (GLOBAL-FUNCTION-NAME BINKY)) (BLOCK BINKY (+ N 2)))) 'NIL) 'BINKY) More examples: (macroexpand '(dotimes (n 10) (print n))) --> (LET ((G243 10) (N 0)) (DECLARE (UNSETTABLE N)) (BLOCK NIL (IF (INT>0-P G243) (TAGBODY G242 (PRINT N) (LOCALLY (DECLARE (SETTABLE N)) (SETQ N (1+ N))) (UNLESS (EQL N G243) (GO G242)))) NIL)) (macroexpand '(dolist (x '(a b c)) (print x))) --> (LET* ((G246 '(A B C)) X) (BLOCK NIL (TAGBODY (GO G245) G244 (SETQ G246 (CDR (THE LIST G246))) (PRINT X) G245 (SETQ X (CAR G246)) (IF G246 (GO G244))))) DEFMACRO symbol lambda-list {declaration | doc-string}* {form}* [Macro] constructs a global macro definition, binds it to symbol, marks symbol as a macro, and returns symbol. defmacro is the macro equivalent of defun. (defvar *people-ages* '((sally 22) (john 21) (tina 56) (eugene 43) (wendy 11) (sam 1))) --> *PEOPLE-AGES* (defun get-age (name &optional (people-ages *people-ages*)) (cond ((null people-ages) nil) ((eq name (car (car people-ages))) (car (cdr (car people-ages)))) (t (get-age name (cdr people-ages))))) --> GET-AGE (get-age 'tina) --> 56 (get-age tina) > Error: Unbound variable: TINA (defmacro age (name) (list 'get-age (list 'quote name))) --> AGE (age tina) --> 56 (macroexpand '(age tina)) --> (GET-AGE 'TINA) Macros have more interesting applications. They are often used to extend the syntax of lisp. (defmacro while (test &rest body) (append (list 'do nil (list (list 'not test))) body)) --> WHILE (let ((n 1)) (while (< n 10) (print n) (setq n (+ n 1)))) 1 2 3 4 5 6 7 8 9 --> NIL (macroexpand-1 '(while (< n 10) (print n) (setq n (+ n 1)))) --> (DO () ((NOT (< N 10))) (PRINT N) (SETQ N (+ N 1))) BACKQUOTE (`) is handy in writing macros. Backquote acts just like quote except that embedded commas (,) cause evaluation: '(spring forward fall back) --> (SPRING FORWARD FALL BACK) `(spring forward fall back) --> (SPRING FORWARD FALL BACK) `(spring forward fall back ,(+ 100 265) times) --> (SPRING FORWARD FALL BACK 365 TIMES) '(spring forward fall back ,(+ 100 265) times) > Error: Comma not inside backquote (list 'spring 'forward 'fall 'back (+ 100 265) 'times) --> (SPRING FORWARD FALL BACK 365 TIMES) Commas can be nested deep within list structure: (setq name 'barney) --> BARNEY (setq occupation 'bozo) --> BOZO (append '(hello) `(mister ,name (the ,occupation))) --> (HELLO MISTER BARNEY (THE BOZO)) COMMA-ATSIGN (,@) within a backquote "splices" its list-valued result into the larger list: (setq fudds-law '(if you push something hard enough it will fall over)) --> (IF YOU PUSH SOMETHING HARD ENOUGH IT WILL FALL OVER) `(in my youth i though that ,@fudds-law -- but now i know better) --> (IN MY YOUTH I THOUGH THAT IF YOU PUSH SOMETHING HARD ENOUGH IT WILL FALL OVER -- BUT NOW I KNOW BETTER) The backquote syntax allows us to write a much cleaner WHILE macro: (defmacro while (test &rest body) `(do () ((not ,test)) ,@body)) --> WHILE (let ((n 1)) (while (< n 10) (print n) (setq n (+ n 1)))) 1 2 3 4 5 6 7 8 9 --> NIL (macroexpand-1 '(while (< n 10) (print n) (setq n (+ n 1)))) --> (DO () ((NOT (< N 10))) (PRINT N) (SETQ N (+ N 1))) Here's another macro-based language extension: (defmacro arithmetic-if (test neg-form zero-form pos-form) `(let ((x ,test)) (cond ((< x 0) ,neg-form) ((= x 0) ,zero-form) (t ,pos-form)))) --> ARITHMETIC-IF (defvar binky) --> BINKY (defvar bonko) --> BONKO (setq bonko 99) --> 99 (setq binky 44) --> 44 (macroexpand '(arithmetic-if (- binky bonko) (* binky 2) (* binky 100) (* binky bonko))) --> (LET ((X (- BINKY BONKO))) (COND ((< X 0) (* BINKY 2)) ((= X 0) (* BINKY 100)) (T (* BINKY BONKO)))) (arithmetic-if (- binky bonko) (* binky 2) (* binky 100) (* binky bonko)) --> 88 Unfortunately, we have problems if we use the variable x: (defvar x) --> X (setq x 10) --> 10 (arithmetic-if (- x 3) (* x 2) (* x 100) (* x x)) --> 49 (macroexpand '(arithmetic-if (- x 3) (* x 2) (* x 100) (* x x))) --> (LET ((X (- X 3))) (COND ((< X 0) (* X 2)) ((= X 0) (* X 100)) (T (* X X)))) The solution is gensym: GENSYM &optional string-or-number [Function] creates and returns a unique uninterned symbol. If string-or-number is given, it will be used in the name of the new symbol. (gensym "foo") --> foo288 (gensym "FOO") --> FOO289 (defmacro arithmetic-if (test neg-form zero-form pos-form) (let ((var (gensym))) `(let ((,var ,test)) (cond ((< ,var 0) ,neg-form) ((= ,var 0) ,zero-form) (t ,pos-form))))) --> ARITHMETIC-IF (macroexpand '(arithmetic-if (- x 3) (* x 2) (* x 100) (* x x))) --> (LET ((G293 (- X 3))) (COND ((< G293 0) (* X 2)) ((= G293 0) (* X 100)) (T (* X X)))) (arithmetic-if (- x 3) (* x 2) (* x 100) (* x x)) --> 100 The function MACROEXPAND-1 can be used to expand only one level of macro definitions: MACROEXPAND-1 form &optional environment [Function] returns the result of expanding form oncewithin environment. Returns the expansion and a second value, t if the form was a macro call and nil if it was not. (defmacro fortran (operator &rest args) (case operator (if (cons 'arithmetic-if args)) (do (append '(do) (new-line-num) args)) (go (cons 'go args)))) --> FORTRAN (fortran if (- x 3) (* x 2) (* x 100) (* x x)) --> 100 (MACROEXPAND-1 '(fortran if (- x 3) (* x 2) (* x 100) (* x x))) --> (ARITHMETIC-IF (- X 3) (* X 2) (* X 100) (* X X)) (MACROEXPAND '(fortran if (- x 3) (* x 2) (* x 100) (* x x))) --> (LET ((G304 (- X 3))) (COND ((< G304 0) (* X 2)) ((= G304 0) (* X 100)) (T (* X X)))) We can write a recursive version of macroexpand that expands all macro calls: (defun macroexpand* (form) (if (atom form) form (let ((expansion (macroexpand form))) (if (consp expansion) (cons (macroexpand* (car expansion)) (macroexpand* (cdr expansion))) expansion)))) --> MACROEXPAND* (macroexpand* '(cond ((> x y) 'x-higher) ((< x y) 'x-lower) (t 'same))) --> (IF (> X Y) (PROGN 'X-HIGHER) (IF (< X Y) (PROGN 'X-LOWER) (IF T (PROGN 'SAME) NIL))) (macroexpand '(cond ((> x y) 'x-higher) ((< x y) 'x-lower) (t 'same))) --> (IF (> X Y) (PROGN 'X-HIGHER) (COND ((< X Y) 'X-LOWER) (T 'SAME))) Macros can be used to extend the language in radical ways: (defmacro deftwinkie (name arg-list body) `(defun ,name ,arg-list (format t "~% I like twinkies yes I do,") (format t "~% I like twinkies how 'bout you?") ,body)) --> DEFTWINKIE (macroexpand-1 '(deftwinkie doubler (n) (* n 2))) --> (DEFUN DOUBLER (N) (FORMAT T ~% I like twinkies yes I do,) (FORMAT T ~% I like twinkies how 'bout you?) (* N 2)) (deftwinkie doubler (n) (* n 2)) --> DOUBLER (doubler 23) I like twinkies yes I do, I like twinkies how 'bout you? --> 46 But overuse of macros can cause problems: (defun age-twice (name) (list (age name) (age name))) --> AGE-TWICE (age-twice tina) > Error: Unbound variable: TINA (age-twice 'tina) --> (NIL NIL) (defun age-twice (name) (list (get-age name) (get-age name))) --> AGE-TWICE (age-twice 'tina) --> (56 56) Note also that macros cannot be passed to MAPCAR, FUNCALL, etc: (mapcar #'age '(tina john wendy)) > Error: Undefined function: AGE (mapcar #'get-age '(tina john wendy)) --> (56 21 11) When should you use macros? [this section borrows from Paul Graham's ON LISP, Prentice Hall, 1994 -- THE book to read about lisp macros] Here are some of the nifty things you can do with macros: - Argument transformation -- e.g., the SETF macro, which picks apart its arguments before evaluation. - Conditional evaluation of arguments -- like IF, WHEN, COND, etc. - Multiple evaluation of arguments-- like DO, WHILE, etc. - Use of the calling environment -- a macro expansion replaces the macro call in the lexical scope of the call -- hence it can use and change lexical bindings in ways that functions can't. For example, the behavior of the macro: (defmacro foo (x) `(+ ,x y)) depends on the binding of y where foo is called. Graham notes that "This kind of lexical intercourse is usually viewed more as a source of contagion than a source of pleasure." - Reduction of function call overheads -- there is no overhead associated with macro calls. By runtime, the macro call has been replaced by its expansion. - Computation at compile time -- you can sometimes move a *lot* of computation to compile-time, reducing the runtime computation to a minimum. - Integration with Lisp -- sometimes you can write macros that transform problems, in a higher-level language of your own design, into simple Lisp. The downside of macros: - Functions are data -- they can be passed as arguments, returned from other functions, etc. None of this is possible with macros. - Clarity of source code -- macro definitions can get hairy. - Clarity at runtime -- macros can be harder to debug, you can't trace them (because they're gone by runtime) and stack backtraces are less informative when you use lots of macros. - Recursion -- an expansion function may be recursive, but the expansion itself may not be. By combining the power of closures with the power of macros I created a utility called DEFUN-CLOSED. (defmacro defun-closed (function-name (&rest lambda-list) (&rest closed-symbol-list) &body body) (let ((closed-symbols (mapcar #'(lambda (item) (if (listp item) (car item) item)) closed-symbol-list))) `(let ,(mapcar #'(lambda (sym) (if (listp sym) sym (list sym nil))) closed-symbol-list) (let ((closure-list (list (function (lambda ,lambda-list ,@body)) ,@(mapcar #'(lambda (sym) (let ((arg (gensym))) `(function (lambda (,arg) (setq ,sym ,arg))))) closed-symbols)))) (setf (symbol-function ',function-name) (car closure-list)) (mapc #'(lambda (sym closure) (setf (symbol-function (intern (concatenate 'string "SET-" (symbol-name ',function-name) "-" (symbol-name sym)))) closure)) ',closed-symbols (cdr closure-list))) ',function-name))) --> DEFUN-CLOSED DEFUN-CLOSED is just like DEFUN except that it takes an extra argument between the argument-list and the body of the definition. This list, called the CLOSED-SYMBOL-LIST creates variabls that persist across successive calls to the defined function. In addition, symbol-setting functions are defined, named SET--. DEFUN-CLOSED Examples (defun-closed calc () ((a 0) (b 0)) (+ a b)) --> CALC (calc) --> 0 (set-calc-a 2) --> 2 (calc) --> 2 (set-calc-b 3) --> 3 (calc) --> 5 (defun-closed accumulator (&optional n) ((total 0)) "Returns the sum of all values passed to accumulator so far." (if n (setq total (+ total n))) total) --> ACCUMULATOR (accumulator 4) --> 4 (accumulator 6) --> 10 (accumulator) --> 10