Friday, August 12, 2011

I am not afraid to admit that I've used Lisp for real work

[Really long post, but I don't see how to break it into smaller pieces. I wrote it a few weeks ago. I didn't want to post it at that time because I wrote some negative things about certain Lisp implementations. I took out a few of the negative comments, but most of them are still there. I stand by my opinions. You're free to have your own opinions, just like me, but I don't publish comments with swear words. I also don't publish comments that are uncivil or completely uninformative.]

As is the case with most programmers, I'd heard of Lisp, but didn't take it seriously because it was always presented as a dead language. Many of those comments came from former Lisp users. Lisp was just not up to the standards of "modern" languages, like Java, C++, Python, Ruby, (add your favorite language). Lots of nostalgia, but they'd made peace with the death of Lisp the way you make peace with the death of a pet dog when you're in middle school.

As is also the case for many programmers, I came upon the writings of Paul Graham and decided that Lisp might be worthy of further exploration. Then I read that the only app anyone admits having written in Lisp was later rewritten in Python. (That's only a slight exaggeration.)

It's hard to come upon examples of the use of Lisp the way you can find examples of Ruby, Python, or even Tcl.

Let me proudly say that I'm not afraid to write programs in Lisp. My situation is different from, say, a web developer for a Fortune 500 corporation. I do numerical programming. I get paid to do simulations, statistical analysis, and a variety of other types of numerical analysis.

I have one of the best jobs in the world. I am free to choose my programming language. It makes no difference if I'm the only programmer in the world using a particular language. Most of the programs I write are less than a few thousand lines, and most of the time, they are almost independent of all the other programs I write. In other words, what matters is that the program I write for a specific task does its job for that task and that task alone. Yep, I do scripting.

Which languages do I use? If you have read other posts on this blog (making you part of a very small group) you know R is what I use most often. Yet I love Lisp and plan to do a lot more Lisp programming going forward. I would not be surprised if five years from now Lisp was my main programming language.

Lisp is a great language and you too can trust it if you write numerical programs of 5000 lines or less. You can probably trust it for other tasks as well but I don't have such an experience with Lisp. Please tell the world that Lisp is a great language for numerical/statistical analysis. Those who dismiss it are wrong. There's absolutely no reason to not give it a trial. See what libraries are available. Look at some code written by others. It never hurts to learn another language, and if you have used only Matlab, you will learn a hell of a lot about programming even if the only thing you do is read The Scheme Programming Language without writing a single line of Lisp code. You have a lot to gain and nothing to lose. I guarantee it.*

Now that I've given a little background, I'd like to discuss the question of which Lisp to use. There are a number of alternatives in the Lisp world. The choice can be confusing to beginners, which would have included me not that long ago. I will briefly discuss four of the available options: Common Lisp, Scheme, Clojure, and newLISP.

Common Lisp: This one is often described as the "industrial strength" Lisp. In many cases, someone will just say "Lisp" when in fact they are referring to Common Lisp. My assessment of Common Lisp is that it's what you'd have if you start with Java, take out most of the good stuff, add on some Lisp features in the clumsiest possible way, and leave changes to a political process that ensures change will never happen.

Common Lisp is sort of the Oldsmobile of programming languages. It was good relative to the alternatives in the mid-80's, but then it stagnated, leading to its downfall.

There are some advantages. It's the only Lisp that's ever been popular in the enterprise. There's a lot of documentation available. There are a lot of libraries available. Some of the compilers have the potential to produce really fast code. There are, possibly surprisingly to the outsider, a lot of programmers who have used Common Lisp. That helps to the extent that you need to interact with other Lisp users, say to ask questions.

The community is a serious drawback. While most of the community is great, they haven't figured out how to control a vocal few.

My opinion: Stay away from Common Lisp as a beginner. The user community is stuck in the disco era. You can always try it later if you want. If you try only Common Lisp you will agree with those who say Lisp is not a suitable alternative to today's more popular languages. It had its chance to be popular, it failed spectacularly, and it failed spectacularly for a reason. If you have a beard and 15 years of Unix experience, you will probably feel right at home, so my advice doesn't apply to you.

Scheme: My first use of Scheme to any degree was going through SICP. Absolutely awesome experience.

I've written elsewhere about the big problem with Scheme. It's elegant, but the core language is too small to do anything useful, and the extensions are of exceptionally low quality, with some of the worst documentation of any product in the western hemisphere.

Scheme is an excellent language for teaching programming. The Scheme developers do not have an incentive to put together high-quality, properly documented language extensions. This is what you will encounter if you read the documentation on a Scheme extension:

foo [option] name code

foo transforms name into code. You can set options as desired.

Section 5 of the Scheme specification prohibits the use of informative examples when documenting functions. The 80th time I had to look at source code to figure out what an extension did was when I gave up on Scheme.

The developers of Scheme extensions are happy when they can say, "It is possible to do X" for any choice of X. To be useful, they'd need to be able to say, "It is possible to do X in a reasonable amount of time with a reasonable amount of effort." You sure as heck don't want to be relying on a piece of code and then find out that it's not doing what you thought it was, and have no idea how to fix the problem. That's when you start thinking about the reaction of your family if you get fired.

And no, Racket is not different in that regard, the extensions are a complete mess and the examples, if they exist at all, are unlikely to help much. This is not a criticism of Scheme or Racket. They are intended to be used as tools for teaching and research. I'm fine with that. There are other choices available. If you are learning Lisp do not get frustrated that Scheme/Racket can't do what you need. That does not mean Lisp is a bad choice of language.

Clojure: This is one awesome language. It's a modern Lisp. Not only is the language awesome, but the creator, Rich Hickey, has a lot of interesting things to say. You can tell that he has been in the trenches for a long time doing real programming, yet he is excited to incorporate ideas from the academic community.

Oh, Clojure also runs on the JVM, meaning you have access to everything provided by the king of enterprise programming languages. There are several good Clojure books out already, and more are on the way. There is serious upside both in terms of the language and the prospects for a career writing Lisp programs.

There are at least two downsides. The first is that using Clojure almost guarantees you will be bringing in some Java. That means many of the bad things about Java apply to Clojure as well. With time, Clojure is likely to develop its own libraries, which will minimize the Java drawbacks. The second downside is that the development tools are not mature, as you should expect with such a young language.

newLISP: Ultimately, I have come to like newLISP the best. newLISP is unmatched in the Lisp world in terms of documentation. After dealing with Common Lisp and Scheme documentation, I was shocked at the quality and quantity of examples that come with newLISP. You can learn to do everything the language does with little difficulty.

Another attractive feature of newLISP is that Common Lisp programmers criticize it for failing to meet their definition of being a "real" Lisp. That's a good sign, because Common Lisp has failed, so if you want to put together a language programmers actually want to use you better do some things differently from Common Lisp. newLISP lets you get work done quickly and correctly, and that's the only reason I've ever used a programing language.

As a scripting language, newLISP is not a competitor to either Common Lisp or Clojure. It is not a speed demon, but if you're using a scripting language, you obviously should not expect the speed of C or Java. It does quite well in terms of speed relative to other scripting languages I've used including Tcl and Python.

For most Linux distributions you will have to compile it yourself. However, that is easy (almost trivial - just follow the directions) and extremely fast (measured in seconds rather than minutes).

The bottom line: If you want to get started with Lisp for real world projects and have programming experience, install newLISP and try it out. The documentation on the website will definitely help you to get going quickly.

If you want to learn to program, install Racket, and use drracket with SICP. Any Scheme will also do.

If you want something suitable for the enterprise, get one of the Clojure books and give that a try.

Although I don't have any recommend uses for Common Lisp, there are many resources available if you want to go that route. CLISP is a good choice, but keep in mind that it does not come with a compiler. I will add that Land of Lisp is a very, very good book (and recent to boot) that will be one of the best purchases you ever make.

The decision of which Lisp is not as big as it might seem. You can carry a lot of your knowledge from one to the other. The best strategy may well be to study all of them. Whichever you choose you will be happy with your choice - even if that choice turns out to be Common Lisp.

*I'm an anonymous blogger, so don't expect my guarantee to mean much.

16 comments:

Anonymous said...

One of the tangible benefits of Lisp is the seriously bright people who hang around the water cooler. I've loitered in both the CL (SBCL) and Clojure communities, and always have to stretch to keep up.

I wouldn't worry so much about personality issues in the CL crowd; killfiles work as well there as anyplace else.

Orest Ivasiv said...

Hi,
Nice post.
But, what Lisp do you use daily, newLisp?

lmf said...

@Anonymous: Great point, one of the advantages of Lisp and more generally functional programming is that the programmers are so much smarter and more knowledgeable than me. It's those who have a passion for programming that get into functional programming.

The part about the CL community was only intended for beginners. I actually kind of like CL (not that it came through in the post), but I'm not comfortable with having CL or its community represent Lisp to the world.

@Orest: Unfortunately I don't yet use Lisp for work daily. I've used both Clojure and newLISP for work. I probably prefer newLISP today. My answer in a year (when hopefully I've been able to ditch all other languages) might be different.

I think the libraries available for the JVM will be critical for bigger programs. On the other hand, newLISP has the best FFI I've ever seen, so it might work out.

Anonymous said...

"...I'd heard of Lisp, but didn't take it seriously because it was always presented as a dead language."

People who don't learn from history are going to repeat it. It is fun writing and playing with new languages. That's why so many "experts" declare that languages like Lisp are dead.

It's great that we have so many choices but don't forget about some of the older stuff. You'd be surprised how relevant it is if you have an open mind.

Florian Over said...

Did you ever tried Chicken Scheme http://www.call-cc.org ?
It's a really nice project, with modern libraries and a small but very active community. Good documentation.

List of libraries:
http://wiki.call-cc.org/chicken-projects/egg-index-4.html

lmf said...

@Florian

I have worked with and continue to work with Chicken, probably more than any other Scheme implementation, as I've gone through SICP.

There is a large quantity of documentation for the extensions, but the usefulness varies a lot from one (egg?) to another. Much of it is a vague statement about what a particular function does, without actually conveying any information.

My last attempt to use Chicken for something other than SICP involved sockets. I don't recall precisely what it was, but I just couldn't figure it out from the documentation. I knew that if I spent an hour figuring it out, six lines of code later I'd be in the same situation with a different function. If this were hobby time it wouldn't be a big deal but losing even one hour of work time to poor documentation is expensive.

This is not a criticism of Chicken, it just didn't work for me. Chicken has a lot of things going for it. I'd encourage everyone to give it a try.

Anonymous said...

Onе time they ԁeveloρed іntо good frіends, they
ωill assume familiaгity, but nοt
much too nοticеably rіght until then.
Sitting ԁoωn there on your counter, it raгely
seеms to be able of theѕe kindѕ of ωοnԁers, but the
first timers and the seasοned сooks alike
will loѵe іncluding thаt customaгy flavoг tо their
popular dishes. The wall panels can bе
introԁucеd in hаllways, doorωayѕ, and shining ѕtraight as a
dance flooring.

Feel freе to surf to my blog poѕt :: Old Stone Oven Pizza Stone Round 14 Inch
my page - Buy Pizza Stone walmart

Anonymous said...

Hi! Someοne in my Fаceboοk gгoup shаred thiѕ ѕite
ωіth uѕ so I came to сhеck іt οut.

I'm definitely enjoying the information. I'm book-mаrking
аnd ωill be twеeting this tо my followегs!
Great blοg anԁ brilliаnt deѕign and stуle.


Take a look at mу webpage augenoperation
my website: augen lasern

Anonymous said...

Hі therе! This articlе couldn't be written much better! Looking at this article reminds me of my previous roommate! He constantly kept talking about this. I am going to forward this information to him. Pretty sure he's
going to have a very good read. Thanks fоr ѕharing!



my websіte :: Chemietoilette

Anonymous said...

I ωould not put it in the fгeezer for a long time, you can put it in the freezer for a couple of hours but І would not put it
in therе fοr a couple of weeks.
Layer bottοm with as much or as little pizza ѕauce as you liκe.
One of the maіn аttrаction to
these cоnvection toaster ovens are thе big brіght dіgital ԁisplау counterѕ thаt keеp time ωhen not in
use.

my ωebpage pizza pan akron menu

Anonymous said...

Howdy this is kind of of off topic but I was wаnting to know if blogs uѕе
WYSIWYG editors οr if you hаve to mаnually сode with HTML.
I'm starting a blog soon but have no coding skills so I wanted to get guidance from someone with experience. Any help would be greatly appreciated!

Also visit my blog post :: Chemietoilette

Anonymous said...

We stumbled oveг hеre by a different ωеb
page and thought I might сheсk thingѕ out.

I lіke what I see so noω i аm following уоu.
Loοk forwaгd to fіnding out about your web page for a second timе.


My blog post; Chemietoilette

Anonymous said...

When аn individuаl decides to make thе transition frоm
eаting meat to being a true ѵegan, many of thе usual thіngs thеу onсе
enjoyеd go out the ωіndow.
Placе sliceԁ tomatoes and Αгugulа on toр of thе Mozzarella.
Τhеу cаn uѕe qualitу ρroduce to maκе
simple and great flavors.

My ωeb sitе - pizza stone

Anonymous said...

I visited many websites but the audio feature for audio songs existing at this site
is actually fabulous.

Here is my website; http://www.thisnext.com/

Anonymous said...

smokeless cigarettes, best electronic cigarette, smokeless cigarettes, electronic cigarettes, electronic cigarettes, e cigarette health

Pablo A. Perez-Fernandez said...

Surprisingly well written article. Learn Scheme as my first college language in my CS classes in the late 1980s. Then, used common lisp late in my undergrad in algorithmic design and analysis. Funny enough, I have started using again under Babel-Org in my finance research mixed with other languages (mainly Python and R) to access their extensive numerical routines. I find Lisp to be the best at dicing and slicing data and structuring my programs.