Extreme software

Some programs are very successful with taking a paradigm to an extreme.

Emacs tries to represent every user interface as text. Even video editing.

Plan 9 tries to use filesystem trees as APIs for all applications and OS services.

It's interesting to abstract insights gained from extreme programs, and modify them.

We can ask: if Emacs is successful using plain text as user interface, what if we use hypertext?

Or: if Plan 9 is successful in using filesystem trees as APIs, what if we use activity streams?

Ouch

Few companies that installed computers to reduce the employment of clerks have realized their expectations.... They now need more, and more expensive clerks even though they call them "operators" or "programmers."

— Peter F. Drucker

TermKit

I happen to think that pushing beyond plain text is one of the most important tasks for programmers today, or as Conor McBride put it:

The real modern question for programmers is what we can do, given that we actually have computers. Editors as flexible paper won't cut it.

So of course I like TermKit:

It brings back memories of Apple Dylan:

It's simple: replace plain text files with collections, and lines with objects, and you get Good Things for free!

For example: Scott McKay describes the DEUCE editor:

The editor for FunO's Dylan product -- Deuce -- is the next generation of Zwei in many ways. It has first class polymorphic lines, first class BPs [buffer pointers], and introduces the idea first class "source containers" and "source sections". A buffer is then dynamically composed of "section nodes". This extra generality costs in space (it takes about 2 bytes of storage for every byte in a source file, whereas gnuemacs and the LW editor takes about 1 byte), and it costs a little in performance, but in return it's much easier to build some cool features:

- Multiple fonts and colors fall right out (it took me about 1 day to get this working, and most of the work for fonts was because FunO Dylan doesn't have built-in support for "rich characters", so I had to roll my own).

- Graphics display falls right out (e.g., the display of a buffer can show lines that separate sections, and there is a column of icons that show where breakpoints are set, where there are compiler warnings, etc. Doing both these things took less than 1 day, but a comparable feature in Zwei took a week. I wonder how long it took to do the icons in Lucid's C/C++ environment, whose name I can't recall.)

- "Composite buffers" (buffers built by generating functions such as "callers of 'foo'" or "subclasses of 'window') fall right out of this design, and again, it took less than a day to do this. It took a very talented hacker more than a month to build a comparable (but non-extensible) version in Zwei for an in-house VC system, and it never really worked right.

Of course, the Deuce design was driven by knowing about the sorts of things that gnuemacs and Zwei didn't get right (*). It's so much easier to stand on other people shoulders...

Secret toplevel

A somewhat arcane aspect of hygienic macro systems is the notion that macro-introduced toplevel identifiers are secret.

An example of this in EdgeLisp:

The macro FOO expands to code that defines a variable X using DEFVAR, and prints its value:

(defmacro foo () #'(progn (defvar x 1) (print x)))

(#' is EdgeLisp's code quotation operator.)

Using the macro has the expected effect:

(foo)

1

But X isn't actually a global variable now:

x

; Condition: The variable x is unbound.

; Restarts:

; 1: #[handler [use-value]]

; 2: #[handler [retry-repl-request]]

; 3: #[handler [abort]]

The name X only makes sense inside the expansion of (foo). Outside, it gets transparently renamed by the hygienic macro system.

(In EdgeLisp, a UUID is attached as color (or hygiene context) to the identifier.)

Why the secrecy of toplevel variables? Well, it's simply an extension of the notion that identifiers that introduce new variable bindings (such as those in a LET) need to be treated specially to prevent unhygienic name clashes between macro- and user-written code. This secrecy completely frees macro writers from having to care about the identifiers they choose.

(Discussion of this topic on the R7RS list: Are generated toplevel definitions secret?)

The why of macros

Good analysis by Vladimir Sedach:

The entire point of programming is automation. The question that immediately comes to mind after you learn this fact is - why not program a computer to program itself? Macros are a simple mechanism for generating code, in other words, automating programming. [...]

This is also the reason why functional programming languages ignore macros. The people behind them are not interested in programming automation. [Milner] created ML to help automate proofs. The Haskell gang is primarily interested in advancing applied type theory. [...]

Adding macros to ML will have no impact on its usefulness for building theorem provers. You can't make APL or Matlab better languages for working with arrays by adding macros. But as soon as you need to express new domain concepts in a language that does not natively support them, macros become essential to maintaining good, concise code. This IMO is the largest missing piece in most projects based around domain-driven design.

Hygiene in EdgeLisp

EdgeLisp is chugging along nicely. I'll soon do a proper release.

I just implemented a variant of SRFI 72, that is a hygienic defmacro. (I wrote two articles about SRFI 72: part 1 and part 2).

(defmacro swap (x y)
 #`(let ((tmp ,x))
     (setq ,x ,y)
     (setq ,y tmp)))
nil
(defvar x 1)
1
(defvar tmp 2)
2
(swap x tmp)
1
x
2
tmp
1
(swap tmp x)
1
x
1
tmp
2

Compiling, linking, and loading in EdgeLisp

EdgeLisp now has first-class object files (called FASL, for Fast-Loadable) and a primitive linker:

Let's compile a FASL from a piece of code:

(defvar fasl (compile #'(print "foo")))
#[fasl [{"execute":"((typeof _lisp_function_print !== \"undefined\" ? _lisp_function_print : lisp_undefined_identifier(\"print\", \"function\", undefined))(null, \"foo\"))"}]]

(Note that EdgeLisp uses #' and #` for code quotation.)

The FASL contains the compiled JavaScript code of the expression (print "foo").

We can load that FASL, which prints foo.

(load fasl)
foo
nil

Let's create a second FASL:

(defvar fasl-2 (compile #'(+ 1 2)))
#[fasl [{"execute":"((typeof _lisp_function_P !== \"undefined\" ? _lisp_function_P : lisp_undefined_identifier(\"+\", \"function\", undefined))(null, (lisp_number(\"+1\")), (lisp_number(\"+2\"))))"}]]

(load fasl-2)
3

Using link we can concatenate the FASLs, combining their effects:

(defvar linked-fasl (link fasl fasl-2))
(load linked-fasl)
foo
3

Compile-time effects

Fasls keep separate the runtime and the compile-time effects of code.

For example, a macro definition returns nil at runtime, and does its work at compile-time (edited for readability):

(compile #'(defmacro foo () #'bar))
#[fasl [

execute:
(typeof _lisp_variable_nil !== "undefined" ? _lisp_variable_nil : lisp_undefined_identifier("nil", "variable", undefined))

compile:

((typeof _lisp_function_Nset_macro_function !== "undefined" ? _lisp_function_Nset_macro_function : lisp_undefined_identifier("%set-macro-function", "function", undefined))(null, "foo", (function(_key_, _lisp_variable_NNform){ lisp_arity_min_max(arguments.length, 2, 2); return (((typeof _lisp_function_Ncompound_apply !== "undefined" ? _lisp_function_Ncompound_apply : lisp_undefined_identifier("%compound-apply", "function", undefined))(null, (function(_key_){ lisp_arity_min_max(arguments.length, 1, 1); return (((new Lisp_identifier_form("bar")))); }), ((typeof _lisp_function_Ncompound_slice !== "undefined" ? _lisp_function_Ncompound_slice : lisp_undefined_identifier("%compound-slice", "function", undefined))(null, (typeof _lisp_variable_NNform !== "undefined" ? _lisp_variable_NNform : lisp_undefined_identifier("%%form", "variable", undefined)), (lisp_number("+1"))))))); }))) ]]

Land of Lisp - The Music Video!

Minimal and sleek / but still so clever you'll freak.

Via Book Review: Land of LISP, by Conrad Barski via @zooko.

Unlimited number of runtimes

Because Lisp macros are written in Lisp, there's a runtime at compile-time. (See this previous post for more.)

Some Lisp compilers produce two separate object files for a .lisp file: A FASL file that contains the runtime effects, and a CFASL that contains the compile-time effects (such as macro definitions).

But why stop at two object files? A single file could in fact produce any number of runtimes (FASLs).

One example where this would make sense is documentation: Imagine a DEFDOC macro (for documenting variables), whose effects take place at documentation-time:

(defvar x 1)

(defdoc x "A cool variable.")

DEFDOC registers the documentation string "A cool variable." with X in some table, so that it can be looked up.

The FASL would contain (defvar x 1), and the DFASL would contain (defdoc x "A cool variable.").

Now it's up to the programmer to decide when and if to load the DFASL: in the development environment, one would always load documentation-time, but for a packaged application maybe not. There one would only ship the FASLs, not the CFASLs and DFASLs (unless the application is intended to be programmed by users).