Why do I still use newLISP?

One of the big* unanswered questions in the world today is why I use newLISP if I am using Clojure. I use R because that's what I've been using for years, I use Fortran because I need speed, and I use Clojure because I want Lisp. So what does newLISP bring to the table when I'm already using Clojure?

Here are a few thoughts:

1. Java is heavy. For example, I work with sockets, and Clojure/Java is overkill for that.

2. You sometimes need an understanding of the Java classpath and all that jazz, and I don't possess that understanding. I can make it work (so far) but I'm not a Java developer and it gets to be a nuisance. The Java build system is a perfectly acceptable reason to avoid Clojure altogether.

3. Another is here:
http://stackoverflow.com/questions/840190/changing-the-current-working-directory-in-java
http://stackoverflow.com/questions/3921744/how-do-i-change-directory-in-command-line-with-clojure


I'm sure you can do what you need to do in Clojure, but I don't want the headache. With newLISP, it's straightforward, and I don't have to learn a new approach for no particular reason.

4. It's easy to call C libraries from newLISP (some of the time anyway). It hasn't been a pleasant experience when I've tried to call C from Java in the past. I'm well aware of SWIG, but SWIG is hardly a fun experience, it's just less painful than the alternatives.


I could add more examples, but the reasoning would be the same. It's partially a matter of scripting language vs full-blown programming language, and partially a matter of JVM vs native platform. I'm probably not going to use newLISP for a big simulation, but there are times that it is the best choice.

I'm not claiming Clojure or any other Lisp can't do these things. Further, I could use Python, Ruby, or Perl for all of my scripting needs. I just find newLISP to be the best solution for me.

* Definitions of big may vary.

Well, that was fast...

I just compiled newLISP on a quad-core machine. It took...4 seconds!

This is a case where it's actually faster to compile than to install from a repo.

Update: Thanks to the comment below, I can do it even faster:

1. Download the newLISP source:
http://newlisp.nfshost.com/downloads/
2. Extract the archive.
3. cd into newlisp-x.x.x
4. ./configure-alt
5. make -j 4
6. (As root) make install

Steps 4-6 took about 10 seconds.


[Deleted previous, now irrelevant steps]

I look forward to using the D programming language in the future

Unfortunately, I don't know that I'm going to spend a lot of time on it right now. In many ways it's like Racket or Scheme. It's a solid language but it's just not at a point where it's reasonable to try to use it for work.

It needs a better website and better access to the outside world, i.e., more libraries. It's "sort of" compatible with C - dealing with C header files is a lot of work. If I want to install an extension in R or Python or some other language (in R they're called packages), they're easy to find and easy to install and get working properly. That's not really the case with D. Heck, it's hard to even tell if individual projects (most of which are partially done at best) are still active.

Hopefully these things will get worked out. I doubt that they will, unless a large corporation decides it has a reason to invest in D development, but I'll give it another try in a year and see if I feel more comfortable with it.

Yes, D is a better C++, but I'm not sure I'm looking for C++. For now I'm more than happy to push forward with my combination of Clojure, R, Fortran, and newLISP. Lisp is just more compatible with my brain.

Languages to learn over the next couple of years: Scala, OCaml, and Haskell. Other possibilities: Go, Factor, Forth, Pure. Even if I don't use a language for real tasks, it's never a bad thing to pick up some new approaches to old problems.

64-bit Firefox builds for Linux

Now that I'm running 64-bit Linux, the one problem I was bumping against was getting updates for Firefox. They've got their new schedule of releasing frequently. Some distros aren't keeping up - and probably for good reason, if they care about stability.

It was easy to find the nightly builds. That's actually pretty slick, with a new update coming through each day automatically. Well, it's slick unless you use your machine for work, in which case you don't want things to break. Oh, also, many of your extensions won't work. Not really what I'm after.

For some reason, Mozilla only provides 32-bit Linux builds on their download page. It will show a 32-bit download link even if your OS is 64-bit. After much searching, I found the 64-bit downloads:

ftp://ftp.mozilla.org/pub/mozilla.org/firefox/releases/

{This is probably something that everyone else already knows. I didn't. This way I can find it in the future.}

CUDA 4.0 on Scientific Linux 6.1

Edit: Further experimentation has indicated that you might run into trouble if you have multiple OSes on your machine. You need to boot with "rdblacklist=nouveau" appended to the line starting "kernel". That is done automatically if SL has control over Grub. If you use another OS that controls the boot manager, you will have to figure out how to make the change. I did battle with this on a machine where Mint, using Grub 2, controls booting.

I don't know much about Grub 2, so I solved the problem by reinstalling Grub from SL. For some reason, in that case, SL assigns a timeout value of 0 seconds, then chainloads the Grub 2 boot menu. It took me a while to figure out what was going on...


-- Original post follows --

I installed the nvidia driver as described in this post:

http://scientificlinuxforum.org/index.php?showtopic=15

using the command

yum --enablerepo=elrepo install kmod-nvidia
No problems after a reboot. I then installed the CUDA toolkit and SDK as described in the documentation. I got an error saying -lcuda could not be found when building the code samples. The solution was to make a symbolic link (as root) using

ln -s /usr/lib64/nvidia/libcuda.so /usr/lib64/libcuda.so

I then did "make" to build all the samples. I ran about half a dozen of them without any problems.

I've had my share of problems setting up CUDA. I'm happy it was so easy.

Calling a C shared library from D

As I said at the end of my post on C++ vs Fortran, I think D is a better language than C++. My preferred language is Lisp (newLISP for little jobs and Clojure for the heavy stuff). The more I play with D, though, the more I like it.

As good as the language is, there are some obvious problems - the website needs to be redone by a professional web designer, and the available libraries on dsource should be better organized so it's easier to find what you're looking for, dead projects thrown out or put into their own area, and a clear separation between the D1 and D2 projects. Even better would be to give some guidelines to contributors, and suggestions as to critical projects.

Until a few days ago I had been slowly learning D by reading The D Programming Language and porting C functions to D. I then decided that D is definitely worth having in my toolbox when I want a compiled language. To be practical I need to be able to link to C shared libraries.

This is such an important thing that you'd expect there to be a clear, easy-to-follow tutorial hitting you in the face as soon as you go to the website. Not really. It's not difficult at all to do. It's just that the information is scattered all around. I don't see any way that someone who does not have C/C++ experience can figure it out. That's a shame because a very good reason to use D is that you want to use a language that provides you with more than just access to the computer's memory (unlike C) and doesn't require a decade to achieve mediocrity (unlike C++).

So here's a simple example. This site shows how to make a .so using GCC. That's  sufficient for our purposes. I called my shared library libmean.so.

On the D side, here is a file (dlinktoc.d) that calls the C function "mean":


import std.stdio;
extern (C) double mean(double a, double b);

void main() {
double a = 4.7;
double b = 7.6;
auto c = mean(a,b);
writeln("The mean is: ",c);
}

This is a typical D program except that I have included the extern (C) line to tell D that mean is a C function. I saved dlinktoc.d in the same directory as I created libmean.so.

The compiling and linking is similar to what you would do with a C program:


dmd dlinktoc.d -L./libmean.so

Running:

./dlinktoc
The mean is: 6.15

C++ vs Fortran

I've been wanting to put together a blog post about Fortran for a while now, but it never made it to the top of my list of priorities. I'm happy to see that someone named eric_t went ahead and put together a really nice page:

https://bitbucket.org/eric_t/modern-fortran/wiki/Home

So, predictably, the C++ mob showed up with their pitchforks to argue against the use of Fortran. (Actually, a lot of them were arguing against FORTRAN, demonstrating their ignorance.)

As someone who has used both C++ and Fortran for numerical programming, it's hard for me to think of a good reason to use C++ for this stuff.

Let me start out by explaining my perspective, which is what rarely happens in programming language debates. I write numerical programs that do statistical analysis, simulations, and various types of numerical analysis. Most of my programs are less than 5000 lines.

I don't know or care about comparisons of C++ against Fortran for 50 developers working with a codebase of a million lines. I work together with only a few others. I have no interest at all in software development. I write programs as necessary to feed myself.

C++ comes with the mother of all learning curves. I'm not talking about the fact that there's no complexity too large to fit into a C++ program (the language is more than happy to accommodate any desired level of complexity). Though of course that is a problem.

I'm talking about the ability to do basic things like linear algebra or writing functions. A Matlab user can convert to Fortran in about 10 minutes. Obviously you won't know very much, but in the time it takes to get used to the idea of compiling a program and specifying types for your variables, you can define two matrices and do a matrix multiplication in Fortran. The language was written for scientific computations. You can take a look at the gfortran manual to see all the built-in goodies that come with a Fortran compiler.

The bare minimum for doing the same exercise in C++ is that you have to get your hands on a library that does what you want. So you have to know about the existence of Eigen or another libary. Then you have to figure out how to call it. Then you have to figure out how to link to an external library. And the documentation is probably not that helpful. There are friendly C++ users for sure. It's just that you won't have a clue what they're telling you to do. They'll be telling you to set up a makefile. They'll be recommending that you look at Boost and talking about how great it is and you'll be all WTF.

All this mess because you wanted to try out C++ to do a simple matrix multiplication. In Matlab it was just a few lines. In C++ it's a month-long project.

And before you use that C++ library, are you really sure you won't end up with a memory leak? Do you even know what a memory leak is? It probably won't matter for your simple matrix multiplication, but it's real comforting to know that you should worry about such things in order to write such a simple program, isn't it?

Now that the numerical version of "Hello, World!" is out of the way, you can get on to real work. You need to start with a tutorial. Not a complete introduction to the language, of course, just a tutorial that will give you a basic understanding of the language.

An excellent tutorial (that has taught me a great deal) is C++ Primer Plus. I just pulled the Fifth Edition off my shelf. It's okay to just read through Chapter 17. You don't even have to read the appendices. The first 1039 pages of the book is enough for now.

You will have worked through Chapter 4, including the section on "Pointers and the Free Store" and "Pointers, Arrays, and Pointer Arithmetic". You'll have learned about "Dereferencing Pointers". You'll get to work through well-designed syntax like (page 308)

int sum(int (*ar2)[4], int size);

or (page 309)

*(*(ar2 + r) + c)

You'll learn about "Memory Models and Namespaces" in chapter 9. The topics include material useful to newbies like "Automatic Variables and the Stack".

Further along (page 677) you'll learn about "Inheritance and Dynamic Memory Allocation".

Maybe you can just learn the language a little bit at a time. Here's the thing. You really can't. You have to know a lot to read code written by others and to use libraries written by others. Complexity is never hidden behind the scenes with C++. You have to understand memory management before you write your first line of code.

You might think you'll just call libraries so you'll be okay. That's not programming in C++, but let's pretend it is. An easy library (written in C) to call is the GSL. You want to solve a system of equations. Here's the function call:

int (* f) (const gsl_vector * x, void * params, gsl_vector * f)

This is an example of "easy" in C++.

Then when you're done with your tutorial, at a minimum, you need to learn about templates and the STL. And, although not strictly necessary, you should pick up some books by Scott Meyers so you can learn about pitfalls of the language.

If you think I'm just some idiot blogger who doesn't know anything about programming, and don't want to take my word that C++ is too complex to be a reasonable language choice, take a look at this blog post:

http://gigamonkeys.wordpress.com/2009/10/16/coders-c-plus-plus/

It was not a group of mediocre developers too lazy to learn a programming language that he interviewed. From my perspective, Ken Thompson's comment is the most relevant:

"It certainly has its good points. But by and large I think it’s a bad language. It does a lot of things half well and it’s just a garbage heap of ideas that are mutually exclusive. Everybody I know, whether it’s personal or corporate, selects a subset and these subsets are different. So it’s not a good language to transport an algorithm—to say, “I wrote it; here, take it.” It’s way too big, way too complex."

If you can program in Matlab, you can program in Fortran. The same is not true for C++. The complexity is not optional: you just can't interact with the outside world without bumping into it.

The advice to use C++ goes something like this:
1. Spend ten years learning to write C++ programs.
2. Start writing all the programs that you could have been writing ten years ago using Fortran.

I understand what these C++ developers are going through. If I invested ten years of my life learning nearly 1% of the C++ language, I would be upset to find out that those ten years were wasted. I'd be looking for some way to make it feel as though I didn't flush ten years of my working life down the toilet.

There are better choices. For many things, Fortran is an excellent choice. It even has recursive functions and lets you specify that a function is pure. It's not Lisp but that does offer a bit of comfort to someone like me.

If you like the object oriented side of things, Java has closed the speed gap a lot (not that I like Java, but being a better choice than C++ is a low standard). There's also D and Objective-C. The more I use D the more I like it. You really should have your head examined if you have the choice between D and C++ and you choose C++ because you think it's a better language. There might be a different reason to use C++ (like, you're too stubborn to admit that your favorite language sucks) but it's definitely not the case that C++ is a better language.

There are languages like Scala (I've played with it a little but not done any real work, so I can't say much about it) and Clojure that have access to the JVM. You can do the performance critical parts in Java if necessary.

LuaJIT2 is really fast, and the upcoming GSL Shell will be based on LuaJIT2 (at least that's my understanding). Even the current version of GSL Shell does well for speed. PyPy is constantly improving as well. Both are excellent, if somewhat incomplete, choices.

C++? No way. There might be a few cases in which C++ is the best choice, but given the time investment required to learn C++, that is a very small set. If you factor in the time to learn the language, write/debug your program, revisit the program later to fix bugs/add features, and then compile and run programs, even pure interpreted Ruby is likely to be a better choice.

Update: I just watched an excellent presentation by Herb Sutter:

http://channel9.msdn.com/posts/C-and-Beyond-2011-Herb-Sutter-Why-C

He definitely makes a strong case for C++. I don't disagree with much of what he said, except that I think he overstates the speed gap for many tasks between C++ and languages like Java, though for the applications he's talking about the difference still matters. Interestingly, for those applications, developer time is almost irrelevant.

He never says C++ has replaced Fortran. In fact, at the beginning of the talk, he specifies that the three languages that can deliver the needed performance are C, C++, and Fortran. Later at that same conference, they talked about D:

http://channel9.msdn.com/posts/Scott-Meyers-Andrei-Alexandrescu-and-Herb-Sutter-C-and-Beyond

Gretl on Scientific Linux

I don't even use Gretl, but sometimes it needs to be available to others. That's why I created Gretl SlackBuilds, and now I have tried to build it for SL.

I pulled the following required packages from the Fedora build at
http://pkgs.fedoraproject.org/gitweb/?p=gretl.git;a=blob_plain;f=gretl.spec;hb=fe69e968cb87ca23907ba147a8071420f702a960

BuildRequires: desktop-file-utils
BuildRequires:    gtk2-devel
BuildRequires:    glib2-devel
BuildRequires:    blas-devel
BuildRequires:    fftw-devel
BuildRequires:    gettext
BuildRequires:    libxml2-devel
BuildRequires:    gtksourceview-devel
BuildRequires:    libgnomeui-devel
BuildRequires:    lapack-devel
BuildRequires:    readline-devel
BuildRequires:    ncurses-devel
BuildRequires:    gmp-devel
BuildRequires:    mpfr-devel
BuildRequires:    gnuplot
BuildRequires: gnu-free-sans-fonts
BuildRequires: bitstream-vera-sans-mono-fonts
BuildRequires: bitstream-vera-sans-fonts





I made sure all of that stuff was installed. Then I did
./configure --with-lapack-prefix=/usr
make -j 4
make check
make install (as root)

Be sure you have everything in the list above. I thought I did, but missed libgnomeui-devel. It compiled and installed, but I ended up with only gretlcli. It won't necessarily fail if you're missing something.

There are packages available on the Gretl homepage, but not 64-bit, which is why I did the compilation. Now if only I could figure out how to get ACML to work with Gretl...

Rmpi on Scientific Linux 6.1

Update: No need to read my frustrated rant. Here's what I did to get Rmpi working on Scientific Linux 6.1 64-bit. It should also work on Red Hat Enterprise Linux 6.1 64-bit and CentOS 6.1 64-bit, though I've not tried, and would appreciate feedback if it doesn't work:

1. yum install openmpi openmpi-devel
2. If you don't have a working R installation: yum install R-core R-devel
3. From within R: install.packages("Rmpi",configure.args=c("--with-Rmpi-include=/usr/include/openmpi-x86_64","--with-Rmpi-libpath=/usr/lib64/openmpi/lib/","--with-Rmpi-type=OPENMPI"))
4. As root (at the command line, not in R) open /etc/ld.so.conf, add /usr/lib64/openmpi/lib, then run ldconfig.


-- Actual post plus rant follows --

Continuing on with my Scientific Linux trial, when I've got a few minutes to spare, I tried to install Rmpi. Oy. I guess I've been spoiled by Slackware (likely the best strategy is just to stick with Slackware 13.1).

First, I installed openmpi and openmpi-devel.

Then I tried installing Rmpi but got stupid errors about mpi.h not found. Went to Google. Wasted time. Went to Google some more. Wasted some more time. Should not be this dang difficult.

Finally, based on this:
http://www.cybaea.net/Blogs/Data/R-tips-Installing-Rmpi-on-Fedora-Linux.html
I was able to construct the magic incantation:

install.packages("Rmpi",configure.args=c("--with-Rmpi-include=/usr/include/openmpi-x86_64","--with-Rmpi-libpath=/usr/lib64/openmpi/lib/","--with-Rmpi-type=OPENMPI"))

Now hopefully it works. I was going to record here all the stuff that needs to be done to get a working SL 6.1 installation that can be used on other machines. It's turning out to be more work than I anticipated. Can it really be the case that nobody uses Rmpi on RHEL-based distros? Or am I just missing the easy way to do this? I thought Rocks Cluster was based on CentOS. Given the popularity of R, somebody must have done this already.

Edit: I also had to add /usr/lib64/openmpi/lib to /etc/ld.so.conf, then run ldconfig, all as root.

gvim on Scientific Linux

Almost got the software I need. gvim is not gvim or vim-gtk, on Scientific Linux it is given in package vim-X11.

RKWard on Scientific Linux

Update: It's probably a good idea to put the summary up front. I even confused myself when installing on another computer this weekend.

Summary:
1. Install any of the above listed build dependencies that are not on your system.
2. Download the latest rkward source.
3. Unpack the source, enter the directory.
4. mkdir build; cd build
5. cmake .. -DCMAKE_INSTALL_PREFIX=`kde4-config --prefix`
6. make -j 4 (or however many processors you've got)
7. make install (as root)
8. Start RKWard. Click on >> Settings >> Configure RKWard
9. Add /usr/share/kde4/apps/rkward/all.pluginmap under the section titled "Select .pluginmap file(s)"

{The last step may not be needed. You'll know if it is when you open RKWard.}


-- Actual post follows --

As with LyX 2.0, I'm going to have to bite the bullet and build RKWard myself. A Google search for "building rkward" leads me to koji.

The requirements are listed as (omitting rmplibs):

R-devel >= 2.13.1

cmake

desktop-file-utils

gettext

giflib-devel

kdelibs-devel

pcre-devel

 I'm missing giflib-devel and kdelibs-devel on my system, so I install them. I download the rkward source and unpack it.

Following the instructions on this page:
http://sourceforge.net/apps/mediawiki/rkward/index.php?title=Building_RKWard_From_Source
I do the following.

mkdir build; cd build
cmake ..        {I'm not specifying any options}
make -j 4
make install

It works (there's an entry for it under "Education") but not correctly. I get a message "Plugins are needed: you may manage these through "Settings->Configure RKWard"." Unfortunately no such option exists.

I look at the documentation again. I try
cmake .. -DCMAKE_INSTALL_PREFIX=`kde4-config --prefix`
make -j 4
make install

The plugins message still comes up, but now I've got a full menu and the ability to open files. I open >> Settings >> Configure RKWard and the menu is on Plugins. I added /usr/share/kde4/apps/rkward/all.pluginmap.

Now when I open RKWard there are no more error messages and everything works properly.

LyX 2.0 on Scientific Linux

I'm giving Scientific Linux a try. While it's a nice looking distro, I'm disappointed that rkward is not available at all, and LyX is still stuck in the 1.6.x series. I'm trying to build LyX 2.0. This is my experience - for my own reference, but if it helps someone else, even better. Google did not provide directions.

yum-builddep lyx doesn't work. Even though I've got the source repos enabled, and used yumdownloader to download the source rpm, I get the message

No source RPM found for lyx-1.6.10-1.el6.x86_64

It would be nice to have (useful) documentation. I am new to the RHEL-based distros so I have no idea what I'm doing.

My initial ./configure says

** moc 4 binary not found !

** uic 4 binary not found !

** qt 4 library not found !


I'm installing qt-devel. I don't see a collection for KDE or Qt development in the package manager GUI, so that's my best guess.

A search for moc turns up automoc, "Automatic moc for Qt 4". Hopefully that's it.

The top result in a search for uic - which I've never heard of before - is basket. I'm pretty sure that's not what I need. Scrolling through the entire list does not bring up anything helpful. A Google search for "lyx uic" turns up nothing helpful. "qt uic" gives background about the "User Interface Compiler" but not anything that solves my problem. "rhel uic" indicates PyQt4 might do the trick.That's what I try. Doesn't seem like the optimal choice, but if it works, I'll take it.

"Configuration of LyX was successful." Awesome!

make - completes with no errors
make install - completes with no errors

I've got a menu item for LyX. Open it up, it's 2.0, everything works.

Summary (can probably improve on the choice of build dependencies, but I have no time or interest in digging into it further - what I've got works)
1. Install qt-devel PyQt4 automoc
2. Download the latest LyX source.
3. Unpack the source and enter the new directory.
4. ./configure
5. make
6. make install (as root)