Sunday, April 15, 2012

What is Lisp syntax, really, and the use-mention distinction

In a recent comment, John Shutt wrote:
I do agree that the success of Lisp depends quite heavily on the fact that it has no syntax dedicated to semantics. But this means that, to the extent that a language introduces syntax dedicated to semantics, to just that extent the language becomes less Lisp.
After a couple of days of thinking about this on and off, in the context of hypercode/graphical syntax, I came to these thoughts:

I think the cornerstone of Lisp, wrt to syntax, is that the barrier between using a piece of syntax and mentioning it is very low.

When I write (+ 1 2), I use the syntax to compute 3. But when I write '(+ 1 2) I just mention it. Note that the difference between use and mention is just one small sign, the quote '.

Let's take a piece of syntax -- corresponding to (let ((x 1) (y 2)) (+ x y)) -- in a hypothetical graphical syntax:

+ x y

The green border around the variable bindings, and their gray background is meant to indicate that the set of bindings is not just usual Lisp list-based syntax, but rather that we're using a special widget here, that affords convenient manipulation of name-value mappings.

This point is important. The graphical syntax is not equivalent to (let ((x 1) (y 2)) (+ x y)) -- it doesn't use lists to represent the LET's bindings, but rather a custom widget. The LET itself and the call to + do use "lists", or rather, their graphical analogue, the gray bordered boxes.

So it seems that we have introduced syntax dedicated to semantics, making this language less Lisp, if John's statement is true.

But I believe this is the wrong way to look at it. Shouldn't we rather ask: does this new syntax have the same low barrier between using and mentioning it, as does the existing Lisp syntax?

Which leads to the question: how can we add new (possibly graphical) syntax to Lisp, while keeping the barrier between use and mention low?

In conclusion: my hunch is that we can add syntax to Lisp, without making it less Lisp, as long as the new syntax can be mentioned as naturally as it can be used.

Follow-up: Escape from the Tyranny of the Typewriter.

HN discussion


Roly Perera said...

This sounds reasonable. Lisp's obsession with representing everything as lists has never made much sense to me, whereas the idea of being able to reify an arbitrary meta-value into a value of the corresponding object-level datatype makes a lot of sense. Your point seems to be that there is no reason for that datatype to be "List". Sure, you can encode environments as lists, but why would you?

John Shutt said...

I don't care whether lists are the data structure used for the representation. I do care that what is used is a profoundly simple and uniform data structure with full set of accessors; and while we're at it, I like that in Kernel the profound simplicity of the data structure dovetails neatly into the primitive unification device of the binding mechanism, so that the full set of accessors just seems to arise spontaneously.

It seems that cons cells are the most profoundly simple data structure that could be used for the purpose. Their extreme simplicity seems to empower the unification/binding/accessors stuff. When considering alternatives, one does hope to fully grok the advantages of what one is looking for an alternative to.

Harrison Ainsworth said...

Can Lisp-ness not be seen as having a simple sharp technical definition? It is in the interpreter, the Lisp reader. The reader handles only one kind of thing: forms, and the only difference in interpretation (semantics) is all concentrated in the first item of a form. ( '(0 1 2) is really just shorthand for (quote 0 1 2) -- everything can be done with ordinary forms, right? -- that is the spirit of it.)

But the Lisp-reader only sees a string of bytes, it does not know anything about how they are presented -- it does not know that parens are curved or spaces are gaps or lines of chars are wrapped into a column.

Rendering that string of bytes that is a form as shaded boxes is effectively just a kind of syntax highlighting.

And furthermore, that would seem the right direction to go. Lisp normally somewhat offends the basic aim of software engineering: clarity. The most basic general requirement of our tools is that they go where we point them and show us where they are pointing. But by making (if...) look so like (quote...) Lisp is fulfilling that requirement rather weakly: it seems reasonable to say that such different forms should *look*, perceptually, significantly different too.

It looks like there are three main layers: syntax (the underlying structure of the machinery), lexical form (the char-based structure), and presentation (how it is rendered in the UI) . . .