Posts
Tyrannosaurus Regex: Markdown for newLISP
... 
Most commented
Script analysis (8)
Dear Editor (7)
Countdown (6)
1 + 2 = 3 (6)
Time flies (5)
Critical math (5)
Choosing a file on MacOS X (4)
Running newLISP scripts in BBEdit and TextWrangler (4)
Super strings: the basics of newLISP strings (4)
Most viewed posts
Report from the engine room (219)
Macro economics (200)
Duck season: part 1 (147)
Tyrannosaurus Regex: Markdown for newLISP (114)
Regexen (106)
Colour me orange (103)
Rabbit season (94)
Looks OK to me (88)
Syntax this (67)
Recent comments
During testing, I've noticed a few problems with this syntax-painting module. One to watch is that consecutive strings that are not separated by a space are not processed correctly. For example:
(println {like}{this})
won't be handled correctly. It's like the scanner doesn't get the time to switch from string to code then back to string. The solution at the moment is to not write strings like that!
Another - more cosmetic - problem is that some of the reserved words with question marks aren't picked up. For example:
(number? list list?)
should all be matched. The regex that matches them needs more work.
As an artist (a bricoleur to be exact), I'm particularly sensitive to the aesthetic qualities of a language.
newLISP is a beautiful language. And that beauty is more than skin-deep. Its simplicity is what makes it beautiful, and newLISP is simple through-and-through.
I feel another motto coming on:
newLISP: Simply beautiful :-)
m i c h a e l
Using newLISP to write a profiler for newLISP programs :-), meta programming is, where Lisp really shines.
Lutz says we don't need the duplicated my-define:my-define in the macro name...
still testing cookies...
Hi! Slurping is cool. And it's what I used too. I just quite like the word, which is much more descriptive than the usual way of describing it!
And ThisService is cool too, Jesper! If I was currently employed, I'd send you some money :-)
As the writer of the Ruby, Perl and AppleScript versions (and, also, of ThisService itself - hello everyone) I just want to defend my usage of slurping. It's really easy - most of the time you want to run the whole input through a filter, and that might not work well with line-by-line or word-by-word or 512-summat-byte-block-by-block.
It will confuse the least amount of people who just want to get stuff done. Peter Hosey wrote the Python scripts completely independently (I said to comment well and keep STDIN/OUT UTF-8 and that was basically it), and he too chose slurping. I'd like to believe that this confirms my suspicions. ;)
Man, newlisper, this is definitely the work of a true bricoleur.
Somehow, with the letters composed of the characters like this, even Comic Sans can be tolerated ;-)
m i c h a e l
Sorry about the formatting. It's hard to retain the original's formatting given the limited width of the text area...
Since this was written we now have parse-date, so it's much easier:
(date (parse-date "2005-10-16 12:12:12" "%Y-%m-%d %H:%M:%S") 0 "%a, %d %b %Y %H:%M %Z")
Colour me orange
12:53 Sunday November 18, 2007
You might be surprised to learn that - despite the bright orange tint everywhere on this site - I'm a bit of a monochromatic minimalist at heart, preferring dark text on light backgrounds and avoiding bright colours on web pages. However, I've decided that it's time that newLISP code on this site should be displayed in colour rather than in dull grey. (I was partly inspired by Cyril's excellent work on the vim syntax module, which you can read about on the newLISP forum, and by the colour schemes available in the newLISP editor, named after composers.)
Up to now, I've been using Lutz' syntax.cgi program for colouring the code in the downloads section. I thought it would be cool to adapt this to use CSS styling, rather than use the old-school <font> tags. And then I also noticed that this syntax program didn't always process every file perfectly - there's a few known problems mentioned in the file itself. So I thought I'd have a go at a new version of the program.
I've set it up so that there are four different CSS styles you can use for newLISP source code. These are selected by one of four classes:
- c for comments
- k for keywords
- s for strings
- p for parentheses
So the HTML code for marking up a function definition looks like this:
<span class="p">(</span>
<span class="k">define</span>
<span class="p">(</span>encode-backslash-escapes t
<span class="p">)</span>
<span class="c">;</span>
which is a lot of text for a simple task (it was even longer until I abbreviated the class names). I've installed the syntax processor to paint the files on the downloads page, and it's working quite well so far. I admit that painting code this way isn't very quick. But I bet Michaelangelo often said that about painting the Sistine Chapel.
A more interesting problem, though, is how to integrate the code painting with Markdown, which I use for writing the text and which is also running as a comment processor. The problem is that Markdown doesn't provide any syntax or convention for specifying which language a piece of code is written in. The only convention available is to indent text in the source file by at least four spaces. Then such text will formatted using white space inside <pre> and <code> tags. You use this convention for all code listings - HTML, CSS, and any other language, and for anything else that relies on formatting defined by white space.
Obviously we don't want to run a newLISP formatting script on a block of HTML code. But there doesn't seem to be a convention for specifying a language, so there are two choices: detect the language using some form of scanning (or guessing), or stick an indicator at the start of the code-block to specify which language to assume.
I've taken the easy way out. The convention I've used in the latest newLISP version of Markdown is simple: if you want to display a code block but you don't want it processed by the newLISP code painter, put an exclamation mark (!) at the start, on a line of its own. Like this:
!
(def-inline-matcher 'link
(make-inline-scanner
'(:sequence
brackets
(:greedy-repetition 0 nil :whitespace-char-class)
#\(
(:register (:greedy-repetition 0 nil (:inverted-char-class #\))))
#\)))
#'link-match)
So that bit of Common Lisp won't be painted in colour. (To show the exclamation mark, I used a second one, but followed it with a space so that it didn't get processed.) But this bit of newLISP will:
(define (process-source source-code-segments)
(let ((result {})
)
; work through segment list
(dolist (pair source-code-segments)
(set 'start (last (first pair)))
(set 'end (last (last pair)))
; put any white space back in
(while (< cursor start) (print (cursor 1 Txt)) (inc 'cursor))
(set 'type (first (first pair)))
(set 'source-string (slice Txt start (- (+ end 1) start)))
(cond
((= type 0)
(push (highlight-keywords source-string) result -1))
((= type 4)
(push (string {<span class="c">} (escape-html source-string) {</span>}) result -1))
(true
(push (string {<span class="s">} (escape-html source-string) {</span>}) result -1)))
(set 'cursor (+ end 1)))
result
)
)
It's a user-friendly solution. Most of the code examples here are in newLISP, and presumably that's true of most of the comments as well, so it's easier to say when you don't want painted code, not when you do.
The main syntax painting code is in syntax.lsp (the syntax.cgi file loads this and builds an HTML page). I've relied heavily on code by newLISP guru Fanda and newLISP creator Lutz. The heart of the script is Fanda's routine that scans newLISP source and records the character positions where the mode (code, string, or comment) changes. Then this list is used to rebuild a copy of the source in which the different sections are enclosed in <span> tags, and the white space gaps are copied over from the original.
This still requires some testing, and some additions (I haven't included the single-character operators yet, because I'm not sure what form of escaping some of them will need). Please let me know of any problems or improvements. And if you can work out how to choose and apply your own colour schemes as well, please share.
λ posted by newlisper on 2007-11-18 at 12:53:34
Comments on: Colour me orange
from newlisper
During testing, I've noticed a few problems with this syntax-painting module. One to watch is that consecutive strings that are not separated by a space are not processed correctly. For example:
won't be handled correctly. It's like the scanner doesn't get the time to switch from string to code then back to string. The solution at the moment is to not write strings like that!
Another - more cosmetic - problem is that some of the reserved words with question marks aren't picked up. For example:
should all be matched. The regex that matches them needs more work.
...comment #1 on post: 20071118125334 added 2007-11-23 at 17:18:37 comment id: 20071123171837