Why not -1 button?

So apparently google have added a +1 button to the search results. Why not a -1? I'm not sure exactly what this is supposed to achieve - yet more ways for spammers to skew the results? I'm already a bit wary of google giving me my own private view of the internet, I hardly want that increased. And often what you're trying to find is the stuff that doesn't come up on the front page now - it seems this would only make that worse.

Just more clutter I don't need or want which simply slows down the page loading.

They also seem to have fucked up the mail client by making a section of the screen not scroll. Which makes it unusably slow and even uglier than it's already ugly forebear. So I guess I will have to go to basic HTML mode now. Which is a pity because I use the chat thing to find out when drinks are on.

Stuff

Been pretty lazy this week - I seemed to spend too much time reading a few sites I frequent from time to time, mostly about the GFC and some of the local political-media clown-show (they are no longer separate entities). But the picture they paint of the world is pretty bleak so it's really all just a bit of a downer; although i'm not sure if it's just the reading that gets me down or feeling a bit flat in the first place that tends to drive me toward reading it.

So no spare-time hacking this week. I did however prune back the golden rain tree in the back yard yesterday - and given we had a couple of days of sunlight I even got a little red in the face. Always nice to get some sunshine in the middle of winter even when such days are few and far between. I also made that lime cordial last week.

For work i'm hitting some big performance problems on the target platform - partly because I think the customer has some unrealistic expectations, and partly because I didn't do enough research at the time on card performance, or they just weren't up to scratch. Oh well. I presume it's something to do with the EOFY purchase dash as well but buying new hardware has come up as a possible solution. Fortunately things have moved a bit since then so at least buying new hardware should be a big help although it wont solve everything.

I'm also pushing for AMD hardware this time - although the Nvidia hardware has been ok as far as that goes, they've obviously given up on OpenCL (no released 1.1 driver, and their opencl 'zone' hasn't changed in a year) and it doesn't seem like a company that wants my money or deserves any support (even the forums are pretty quiet so it seems i'm not alone - we all get the hint). Expanding ones experience and educating yourself about the alternatives is always a good thing too.

By coincidence AMD just had some marketing event about their heterogeneous computing plans, and Anandtech has a really interesting article on where AMD are going with their GPU/CPU architecture. Looks quite promising, although i'd really like to see a bump in local-store size. Although there is certainly enough there to be useful it is still a bottleneck, and with even more parallelism possible due to the design, the limited global bandwidth will only become more of a bottleneck.

Pity it's still a way off, because a change in architecture of that magnitude will require a different approach for performance, although in general it looks like it will be easier and it will also map well to OpenCL.

Lazerimagez

So I needed a bit of a break from work and other stuff I was working on so I've revisited ImageZ in my spare time over the last week or so. I'm contemplating cleaning it up a little and putting it somewhere other than my own hard drive - it's well and truly alpha-quality but maybe it can go beyond that.

I decided trying to have a global tool context was just never going to work, so I've moved them to being per-image. And now I'm in the midst of redoing the drawing tool - recognising that the only thing that changes with the pen types is what it draws at each paint position. So i've made an interface that creates pens on the fly using shape, dimensions, colour, and fill type. Still not sure where i'll end up with this but it seems a more promising direction.


And as also can be seen, I thought it needed a bit more colour ...

I'd forgotten how much I actually had working quite well - layers of different data types (8 bit, 16 bit, float, grey-scale or RGB), infinite undo, compound selections, and a nice save dialogue. Although there are also some pretty broken parts - all blend modes apart from Normal (I changed the alpha model and haven't updated them), image operations, and being able to load and save multi-layered images (that's a bit of a show-stopper). I know one reason I didn't get far on the latter was that there is no native support for loading/saving float format images.

Unfortunately the rest of the household is all croaking from some illness and I feel like i'm headed toward a bit of man-flu, so this weekend will be a slow one I think.

The Player

Not really worth a screen-shot but I kept poking away at the player code and added some seek support. I've got it working pretty well on 'well-behaved' files, although it isn't nearly robust enough for 'general use'.

I then got totally side-tracked and started working on a Linux DVB interface. I just cut & pasted some of the jjmpeg binding code, and created a new shared library and namespace within the jjmpeg project. The task of binding the ioctl interface doesn't really match that of binding libavcodec but using the same mechanism should suffice. Not sure jjdvb is a great namespace either, but then again it will do. Now I just have to drag some coax across the room to the PC to test it, or try it on another machine ...

As an aside, with all this hacking of late I haven't been doing much else, although at least I got out and mowed the lawn today for the first time in 3 weeks. And fortunately caught some of the very brief sunlight. Also swept the leaves up and turned the compost heap over. But I didn't get to making the lime cordial I had planned to, although I prepared the bottles to keep it in.

(Today was a public holiday, and with Friday off too that was one long weekend of hacking - I think I need a day off!).

The Wall

Just for fun I tried throwing a bunch of videos into one window simultaneously.


And yes, the sound plays ... although it is quite disconcerting and disorienting having all 9 play at once ...

The sources are PAL-format digital recordings transcoded to x264, i'm scaling them using libswscale and then simply using swing labels to display them. It's clocking up between 100% and 150% CPU usage on the java process (100% == 1 core on Linux). So even though i'm hardly trying to make it quick it's hardly taxing the box (albeit a very fast box).

For some reason the timing code gets messed up if I try to sync to the audio timestamp, but that might just be these videos.

Video Player

So this weekend I wrote a video player using jjmpeg. Yesterday I had the basic video working, and today I got sound working - using OpenAL (JOAL) for output. For the most part even using a fairly simple synchronisation mechanism it works fairly well. I'm letting the hardware run the sound at it's native rate, and synchronising the video to the timestamps in the file (using Thread.sleep() no less!) - not perfect but it's a start.

I also uploaded a new jjmpegdemo's directory to the jjmpeg project which includes a simple audio player that I used to work out OpenAL.

But the Video player is a bit more complex, I've currently got 1 thread demuxing the input, 1 thread each for video or audio stream decoding, and another thread to synchronise the audio and video. I use some of the nice classes from java.util.concurrent to handle the packet and frame queues which means each bit of code is pretty simple. I'm recycling the AVPacket's, the frames and audio sample buffers so once started it has a pretty low GC load.

I'm having some strange problems though - certain files seem to throw the demuxer right out - I get massive corruption in video and audio and it's getting completely broken data frames. These files play just fine in the AudioPlayer above, so I presume i'm doing something incorrect with my threading which is corrupting something along the way. Other files work just fine though so it doesn't seem to be just a simple problem with invalid code - it also affects different containers and codecs inconsistently. Just the sort of bugs I like ...

Update It seems I had too many threads. I've moved calling the codecs from their own thread to the demux thread. Then I have 2 threads for rendering the audio and video separately instead.

Video List

So one thing i've been mucking about with using jjmpeg for is creating a GUI for listing videos ... which seems a pretty basic starting point for doing anything further.



Yeah it's not much to look at so far but one has to start somewhere. I might look at using Piccolo2d as the rendering surface, although I have to determine how to handle virtual items as I do here with the JList. Apart from general fugliness it flashes white whenever you change the view sort, which is quite unpleasant. As can be seen, I hooked it up to mplayer after you double-click a row, just for a laff ...

Under the bonnet it uses jjmpeg of course to scan the files - it's currently generating 128x128 preview images at 1 minute intervals - of which only the first is shown. I have a separate tool to `import' the videos for the moment but I have code lying about to allow dropping of files, so it wont be hard to add. I'm using Berkeley DB - java edition to hold the meta-data and preview images, and i've hooked it up so the DB is scanned in another thread. I use different secondary indices for each sorted view so they are all just as fast (slow?) as each other - this will also let me query by keyword with a little more code. I'm also using SoftReferences to implement a cache of database items. Unfortunately Berkeley DB JE doesn't let you query by record number like the C version, nor read the secondary database keys without also dereferencing to the primary database (i.e. slower than might otherwise be), but judicious use of threads can help alleviate such issues.

In short: it should scale quite well.

Probably ...

beep

Feeling a bit cold and crappy and had a day of franticly little progress at work so I sat down with jjmpeg for a little while tonight to pick some low hanging fruit to make it feel like I'm getting somewhere.

To that end I added audio decoding support. It only requires a couple of functions and field accessors beyond what you need for video, but they needed some extra native functions to make them work. Mainly with decode_audio3 as it takes an AVPacket but may not fully consume it and doesn't update it to indicate this, so you really need to make a copy of it and update the copy's pointers based on the decoding results. At least I did it in a way which requires no allocation activity on the Java side during the decoding loop, I just use the wrapping ByteBuffer's to perform a memcpy and have a single native function to update the data pointer and size on the copy.

I don't have any immediate need for it, but it was easy enough to add and now the day doesn't feel like a total waste.

I'm slowly working toward a loose idea of 'something' that uses this stuff, although I haven't really pinned down what that 'something' might be exactly. Which leaves the options pretty wide open for now.

Sharing GLContexts

Hit a problem yesterday, how to share GL contexts with OpenCL and several GLCanvas objects. i.e. I have a window with a few output windows and they all need to share OpenCL processing.

I eventually found one of the test cases which did the same thing, and I used a similar approach.

• Create a 1x1 off-screen GLPbuffer object using GLDrawableFactory.
  
• Add an event listener to that.
  
• Call display() which will invoke the GLEventListener.init() method synchronously - in which I then create my CLGLContext object.
  
• A few other bits and pieces like using makeCurrent() on the GLContext taken from the CLGLContext() when creating textures to be shared in the OpenCL code.
  

Bit tricky to debug as most mistakes just lead to clue-free segfaults. It didn't help that I forgot to initialise a width/height variable at one point, and until I did everything 'worked' except that creating the output texture always failed with an error which didn't make sense.

jjmpeg - why java?

As a follow-up to the post about using Java and JNI to access ffmpeg, perhaps the more fundamental query is - why use Java in the first place? After all, Java is slow and crappy and nobody uses it anyway and isn't .NET the way to go and all that?

I used to write Amiga BOOPSI classes in assembler for fuck's sake, so why am I now using Java?

For starters Java is not slow - although as with any language you can (un)intentionally make it slower than necessary. Compared to similar systems with the same application support it isn't bulky; at run-time or on disk. The JVM is mature and stable and the garbage collection is reliable and fast.

Machines are also not slow these days - in-fact they are so fast most of the processing power is wasted much of the time. Likewise for memory. Wasted processing cycles and wasted memory bytes are actually an inefficiency, not necessarily something to chime about. I am no longer developing applications for a 1MB system running a multitasking GUI. Nobody is.

I still enjoy writing C, and I am still concerned with performance and efficiency, but I have been using Java for a few years now and am very happy with it - and I continue to be further pleasantly surprised from time to time. I find it puzzling that far more desktop software isn't written using Java - in my experience it compares well in all the important categories and is generally easier to develop for.

For example, performance is usually within a few % of C for normal scalar, single-threaded C. Most programmers don't seem capable of going beyond that type of code anyway - and those that are will find JNI a piece of piss. It will probably require twice as much memory - but this is simply an artefact of the use of a decently fast garbage collector - nothing comes completely free, but with memory expanding so much in recent years this is about the cheapest cost you could imagine for the huge benefit it provides. And I don't just mean no longer needing to track which pointers to free - I never found that particularly onerous although many people are unable to grok it - the GC is also a very fast memory allocator as well. No need for pool or slice allocators and the whatnot.

By the time you add all of the features of a basic JVM runtime to C (or anything else), you have something like GNOME or KDE which are not very small at all, have large memory footprints themselves, and are still not as easy to work with (speaking of GNOME as of some time ago at least, I haven't tried KDE and in any event loathe C++ so am not about to).

Of course, python (or ruby) seem to be the flavour of the month at the moment, but they have their own issues. Usually they are just ugly front-ends to some C libraries or commands and they have the same problems that tcl/tk scripts had - a specific version dependency, ugly gui's, and meaningless error messages from their inevitable crashes. And for all that they're not particularly robust, nor provide a particularly compact memory footprint.

seek to frame

Seeking to a frame using ffmpeg ...

I knew this was a bit of a pain since I'd tried it before, but oh boy - there went my weekend. And i still don't have a 100% reliable solution. Ho hum.

Some of the issues I found with only a handful of videos I have at hand:

1. An mpeg ts which will wont seek via timestamp. Only byte seeking works.
   
2. An avi in which byte seeking never works. Only timestamp seeking does.
   
3. An avi in which the DTS increments forever - so although you can seek by timestamp to a keyframe, you cannot use it to identify specific frames thereafter.
   
4. A mov file in which byte seeking never works.
   
5. A mov file which ends with an EPIPE error rather than end of file. It must be closed and re-opened to perform any further operations.
   

I have something which mostly works now, but I suspect it will never be reliable enough.

jjmpeg - why jni?

I started writing a reply to Michael's comment on the last entry but because I tend to ramble, it ended up so long I thought I'd promote it to a post.

have you considered using JNA? ...
... otherwise gluegen if you want to stay on the JNI road.

I looked at JNA previously some time ago, and found some problems with using it. I can't remember what they were at the moment but I was so displeased with it I know it ruled me out ever bothering with it again. It looks really good on paper but as I chose to write JNI directly at the time (for cross platform code too) there must have been a good reason. FWIW I didn't look into SWIG or any other option either.

For ffmpeg specifically, you need to access random fields of big structures and it would be impractical (or impossible) to map them using jna - many of the fields are private and the public ones are spread out through the structure. So i'd be forced to write a library to define accessors anyway, and then the jna objects to call those, so in the end i'd have to write something twice whereas now i don't even need to write it once (just a simple config file entry, assuming i didn't write a generator for jna - but then there would be no reason to use it).

I tried gluegen because it looks pretty nice and i've had nothing but positive experiences with jocl and jogl, but it's preprocessor and parser just weren't up to the task - the ffmpeg headers are mostly internal headers which have become public, they are not a cleaned-up public/standard api. They contain a ton of cruft that isn't public as well as public stuff that is behind conditional compilation (using expressions) and the like. I tried pre-processing it using cpp -dD (iirc) which preserves the #defines but then the inline code or other stuff threw it and i couldn't even work out which bit of code was the problem from the terse error messages. After giving up on it i found some other tools that might generate a simple/clean enough file to process (e.g. cproto can dump cleaned up types as well as clean prototypes) - but by then i'd moved on.

The perl script is a bit of a mess but most of the binding is automatic. At minimum i only have to write a constructor method for the public class. Accessors and most methods are automatic (once defined in the config file). I only need 2 classes for every wrapped 'object', one auto-generated.

There are a few special cases, but I find JNI pretty easy to use for those - given what it does it's about as simple as could be expected. And having attempted or worked with interfaces for similar purposes in the past I think JNI is actually quite nice. For example .NET's native binding looks really nice on paper too (it's more like a 'built-in' JNA) but there are actually more gotchas because it's trying to automate more - it's good most of the time but can be a real pain when things get complicated.

I'm already spending more time trying to work out how to use the libraries, the binding itself is mostly looking after itself, even if it is still incomplete.

Having said all that .... I realise that I may have made a mistake and there will be outstanding issues yet to resolve. But at least i'm fighting with my own mistakes and not finding the hidden limitations of tools i know little about - which simply makes it a lot more fun for a spare-time project.

jjmpeg rethink

So I had a bit of a rethink about how jjmpeg does things and in short rewrote it from scratch over the last couple of days/nights:

• OBAWO (one big-arse-write-once) Perl script generates bindings for field accessors as well as many methods based on a configuration file.
  
• Use dlopen() to bind to libavformat, libavcodec, libswscale at run-time, in order to avoid linking to an impossibly specific version of libavcodec at compile-time.
  
• Write accessors in C. This will necessarily be a bit slower, but it avoids having to have different Java classes for each case. It also means only the 'c' files need to be recompiled for a different platform.
  
• The C member functions do their own 'this' lookups, thus allowing them to be called directly as public interfaces (although right now they're not for various reasons).
  

I've now enough of the libraries bound to allow creating of video files:

It scrolls! (not shown)

So with this and with a bit of Java2D it's pretty easy to start compositing and generating simple video sequences, if one should so desire.

I'm pretty sure I have the lifecycle and memory management sorted, although there may still be bugs there. The AVFrame to AVPlane interface is a bit crappy though.

VideoFileChooser

Just a simple little utility class i'm working on at the moment.


A basic file chooser with a greyscale video preview of selected-format video files. Seems like it could be nice to have; even if it is only black and white, mute and with (possibly) incorrect aspect ratio and frame-rate. I just hacked up the image-preview file requester I made for ImageZ and in-fact it took fewer lines of code.

I also had a go at supporting 32 bit systems with jjmpeg, although I haven't tested it at all yet. However I noticed that the .so file links with a huge pile of stuff from ffmpeg depending on the build options it was created with. So it probably will not be possible to make a generic package for it - fortunately the native library is only a single small c file so it probably doesn't need one. Assuming I don't go and dlopen the libraries manually at least ... which is always an option I suppose.