Why not use Scheme more?

Another observation post, probably not worth your time, but I want to express my thoughts. You've been warned.

I really first started using Scheme when I went through SICP. Well, I didn't make it all the way through, but I learned a lot in any event. When you've got masters like that to show you what can be done with Scheme/Lisp you wonder why everyone isn't using it.

As you can guess, I think I know why not everyone is using it. I primarily do numerical computing, so that is the set of features I consider when evaluating a new programming language. A couple of things stand out when I think about using Scheme rather than R and Fortran.

1. There is a lack of libraries. This is well known, sometimes called "the island problem".

2. There is a lack of data structures. If I want to do matrix algebra, I want to use a language that offers a matrix class as part of the language. I want to use a language that does matrix multiplication as part of the language. There are good reasons for that: less hassle, less chance of breakage when the core of the language changes, less chance that you're relying on an extension written by one individual that will stop working when he gets bored and moves on.

Item 1 is not as true as it used to be. The major Schemes have a lot of libraries available. There are also tools for wrapping C code, so honestly, any lack of libraries more or less reflects the small user base, rather than the other way around. If there were more interest, the libraries would appear quickly. There's just no interest.

Item 2 is still a problem and likely to remain so forever. I'm just not sure that a minor inconvenience is a major hurdle to adoption. Something like a standard set of libraries being available with each implementation would be a solution.

I've reconsidered the situation since discovering Clojure, and IMO the real problem with Scheme (any of them) is the sucky documentation. It stinks. It really, really stinks. {Clarification: Upon reading the post, I was not clear that I'm talking about the developers of Scheme extensions, not the developers of the core language itself. There exists very good documentation for the core language if you're doing the usual things.}

Have the Scheme documentation writers never heard of examples?

I've never seen a community that was so collectively clueless about how to write documentation. And I'm a Slackware user. I don't think anyone has ever complained about the excessive documentation that comes with Slackware.

I might even consider writing some documentation myself, if not for the fact that I can never get anything in Scheme to work, because the documentation sucks so bad. My philosophy is to just move on if I have to work too hard to figure something out. I've come to realize that developers that don't document their work very well are usually not good developers.

Let me elaborate on that point a bit. The software might work as intended, but the underlying code is only part of the project, and by itself has no value. A good design means I don't have to think too much to use it. And when I return to the code in a year, I'll be able to easily figure out what I did and how to extend it.

If I have to work for six hours to figure out how to call a simple function, that's a sign that at some point I'm going to find that my program isn't doing what I thought it was doing. At some point I'm going to say, "WTF? Why was that matrix transposed? It makes no sense." Of course there was no way I could have known that, and there's no way I will ever understand the philosophy driving the development of the library. Then a few seconds later I'll be terrified, "Hold on, if the developer would do something like that, how can I trust the other lines of code that I've been writing for the last two weeks? There's no freaking documentation!" Then I'll move on to a real language, redo everything, and be sure to avoid the mistake of trusting Scheme in the future.

Go to one of the popular Scheme implementations. Just pick one. They're all equally useless. Pick out an extension that does something you need for numerical computing. If there's documentation at all - and there may very well not be - try to use that documentation to write a program. Use all of the important "things" that the extension does. Use those important things in different ways, like you would actually use them in an actual programming exercise where you are actually trying to feed yourself. Can you even figure out how to call all of the functions properly in less than two weeks?

Most of the documentation is just a slopped-together list of functions and a short, content-free description that might (though in most cases probably won't) be sufficient to jog the memory of the developer himself as to what the function should do. There's no chance in hell that a programmer writing a program related to his career is going to spend enough time to figure out how to use the extension. Sure, there are hobbyists who will figure it out, but I'm talking about real work in an environment where money is on the line.

I cannot treat the developers of Scheme as real developers. They're not professionals. Some are, of course, but I'm talking averages here, and the average Scheme developer is of pretty low quality relative to the average Java developer. The average Scheme developer thinks he's much better than the average Java developer because he can use recursion properly and was smart enough to recognize the brilliance of Lisp. Great, but if you measure developer quality in terms of the value of the output in a given amount of programmer time, Scheme developers probably have a negative value, because most of what they do gets in the way of developers interested in making a product that does something of value.

Do I like Java? Not a chance in hell. If I had to choose between Scheme and Java to do something with money on the line, even assuming all the same libraries are available in Scheme and Java, which would I choose? I'd take Java in a second. Scheme could learn a lot from Java about working with beginners.

It's a shame that Scheme has been plagued by crappy documentation. There's no reason that Scheme, on technical merit, cannot be Java. The difference is that Java was written for business applications, and that required documentation, and they got it. The Scheme community has decided that working with beginners is not fun and it has caused Scheme to be viewed as a goofy artifact of the past, rather than the language of the future.