Why is software so far behind hardware?

For various reasons I was trying to get some binaries from some very large newsgroups - so large that I couldn't find any client that would even list all of the messages without hanging or crashing. Interestingly, I found the mail/news application in seamonkey to be one of the better ones - it worked much better than thunderbird, using far less memory, although it still got stuck very often, and that was only with 200 000 headers rather than the 6 000 000+ in the group.

I saw a few scripts/programmes for grabbing stuff in the repository for this machine so I gave them a go. I can't remember all I tried but `brag' seemed to fit the bill and worked ok on a smaller test group. Well, needless to say it just didn't scale. Either the code is bung or tclsh has a big memory leak or simply isn't up to the task. It managed to download all of the subject lines (meanwhile memory use - ~2.9G!). Well my machine wasn't really swapping so big deal eh? But then when matching the headers memory use continued to climb - even faster. It ran out of 32-bit virtual memory ... well I guess I should've been using a 64-bit OS for such a complex task!

Err, no, that's just bullshit. So I hacked up a quick and dirty C version. Has somewhat fewer features but works more or less the same for what I needed. A bit over 400 lines of code in total, only using libc. No smart algorithms apart from a zero-copy 'read-line', no asynchronous reads or other decoupling of net i/o from disk i/o or processing, etc. Total memory used whilst reading and saving the subject headers of said 6 000 000+ group - about 100K (and most of that is from the C library, I think I malloc about 50 bytes, and have 11K of bss). Total CPU used, about 10 minutes real-time, and about 5 seconds cpu time (top consistently showed 0% cpu usage - well I mean as it should, it's entirely CPU I/O bound!). It takes about 3 seconds to scan the 780MB cache of titles with a regex when they're in the buffer cache, about 10 otherwise (using the same amount of memory).

Alas, this is the world we live in. This is why a `netbook' isn't big enough to run `real' applications - it will just be destined to run slow scripts which proxy the heavy lifting (when it is quite capable of doing an awful lot of the heavy lifting itself). It will be kept busy struggling to show adverts instead (even something my 4 core machine has trouble with, so I turn off animated gifs and flash and block lots of advert pics). This is why, after all the advances in hardware, we now have shitter apps that run more slowly - e.g. for all it's cracked up to be, compare google-writer to say, abiword, or calc to gnumeric - with no other benefits like stability, security, and so forth.

I was looking at javafx again just recently. Someone hacked (and I mean hacked) up a sort of 'space invaders' game. Except it has no sound (could be my use of oss4), no barriers, the aliens don't move properly or speed up, and unusable controls. And it jerks around in an unplayable fashion, with tearing and an utterly atrocious frame-rate. Flash et al are pretty much the same to me. Slow, jerky - crappy games. The machine in question is capable of emulating real hardware faster than that - you can play the real space invaders (or for me, the C64 version) with proper sound, control, and 'game' all in it's glorious full-frame-rate goodness with no problems (and so it should, it's emulating a 30 year old computer!), but it struggles with a half-arsed rip-off.

Poor algorithm choices. Bad language/framework/environment choices. Poorly implemented language/framework/environment. In short, poor engineering and system design.

Computer Science lecturers always say `we don't teach computer programming'. And it's pretty obvious - no, no they don't.

(TBH `brag' does more processing than just a regex match on the titles. It tries to find the filenames, and count the parts, and tries not to grab duplicates. And it uses multiple connections to improve downloading of the actual messages. Still, there's no reason it should use anywhere near that much memory, loading all subjects into memory at once is not ncessary.)