This will be my last post to a-hackers-craic!
I've moved all the posts and images to http://www.zedzone.space/blog, although how they look and how they work are still very much work in progress. I had already moved most of http://users.on.net/notzed to http://www.zedzone.space yesterday.
Some things might be broken for a while but everything apart from comments should have already made it across intact.
Byeee.
I didn't really find anything I liked so instead i got myself a domain and a virtual private server to play with. I had originally intended to set something up on my home network via a static address but I never got around to it. Unfortunately getting a vps in switzerland is cheaper than even doing that - and that's with the virtual hardware and unlimited network bandwidth the latter of which is big buckeroonies in australia.
Well I still haven't done anything with it apart from setup apache with a nothing homepage pointing back to my isp hosted stuff, but it's at www.zedzone.space.
I'm still pretty lazy on the whole idea so it might take a while but i'll move my code and blog and eventually add an email to it. I don't particularly like working at that end these days (despite multiple shots at CMS subsystems) - too much security and complexity to worry about - but i'll see how it goes.
Although logging in here to post this did give me pause for thought on getting a bit less lazy about it. Despite never having offering it before blogger just gave me a page to buy and link a custom domain for a-hackers-craic.
Thanks google for once again mining my email to sell me something I already bought!
Next it'll be combat boots, kilts, and beer ... Adelaide Review article which includes bits about me (not sure why i didn't introduce myself as Zed that time, would've been better!).
Update: 15/4/2018. I finally got off my laze arse and spent a wet afternoon playing with the site - i've copied my internode (ISP) pages to the server, fixed some paths, and installed cvstrac for browsing the source (of some of the projects) via the new virtual host code.zedzone.space.
Still to do is the blog.
I haven't seen any patches for my laptop or work computers so far - one because i don't run any av software on microsoft? Ugh, what a fucking mess. I barely use that machine any more anyway, even less so while running that junk.
What's most appalling is that the problem has been sat on for 6 months under "responsible disclosure" while intel continued selling gimped hardware based on bogus benchmarks. And yet the linux and *bsd developers are either only just starting on or working on cleaning up fixes very recently - even intel is still fucking around with microcode updates.
The tech press, apart from The Register (who really broke this story) has been pretty shithouse too. "big nothingburger" for a massive massive security hole that isn't actually fixed by any of these OS kernel patches, merely mitigated. It's a big big serious problem, with big big serious costs, and the handling the disclosure has been an utter disaster - seemingly designed to spread fud about the impacts and costs. Still, if you relied on a single supplier - and particularly a nefarious piece of shit company like Intel - for any serious hardware investment, sucked in I guess? We all have to pay for your stupidity in the end regardless.
Blah.
I was insomniac[sic] late night and bored so I took the latest stack overflow survey. Apart from some pretty strange questions it's pretty clear why nobody reads my blog - no mention of almost any of the technologies I use day to day. Well at least they had the brains to identify C and C++ as two separate languages I suppose.
Some of the "order these in importance to you" questions I just skipped - I didn't care about the topic at all. A minor problem was a bit of confusion with "framework" "language", etc - where would OpenCL fit if they even included it? No CUDA either for that matter. Plenty of questions about advertising. No I don't use an ad-blocker but I turn off javascript which makes the site oh so much faster and kills most adverts as a bonus.
Sleep has been pretty miserable lately - probably a side-effect of overdoing it a bit, but I think i've gained enough weight again for the sleep apnoea to kick in. One example of overdoing it was the last weekend. Hit the beach Saturday afternoon, dropped in at a mates place - drank a carton of (bloody expensive) piss, went to a party (total strangers apart from him), was up all night whilst everyone else was falling asleep, back to his place to pick my bike up - for a couple more, another dip, and finally dropped by the pub on the way home for a fast (very slow!) one. Barmaid thought i was managing well all things considered but maybe that's because I always look exhausted and feel like shit? Nothing like a 36 hour day to get you tired; and then I was awake after only 4 hours of bloody sleep. For a couple of silly reasons I also got sunburnt to a crisp, oops.
Finally kicking into summer mode!
Pretty much recovered from NYE and NYEE drinking but I might take it easy for a couple of days. Waiting for the weather to heat up enough to get back to the beach, did some gardening. The replacement BIOS arrived today so I resurrected my PC too - still seems a bit funny so i'll probably get another one soonish but it works again for now.
I found a laser printer dumped on the side of the road so took it apart for something to do. Surprising amount of screws, custom springs, cogs, and lots and lots of tough plastic. Probably nothing I can do with it but the imaging unit is kinda cool - rotating mirror, a couple of lenses and mirrors. I suppose the more surprising thing is how much technology is placed in what is essentially a throwaway, one-use device. The "consumable" isn't the toner in these things, it's the whole machine; and what's worse is the realisation that the economies of scale mean it couldn't be done any other way. The world is so fucked.
When I wrote the following on the new jjmpeg home page:
Version 3+ is a complete rewrite from earlier versions which have effectively dissapeared from the internet after google code closed down.
I didn't realise just how true it was. Without google code and regular blog updates my projects have basically vanished from the internets - and more so from google search than others. I'm not particularly surprised there is no real interest in the projects themselves but effectively vanishing is a bit weird. Unless you're using some proprietary publishing platform you essentially don't exist.
Well i've got a few weeks off at least anyway; i'll take a break from the computer, watch some cricket, and basically just bum around a bit.
Put enough together to push out a release of jjmpeg.
It ended up 1700 lines of Java, 2000 lines of C, and 300 lines of Perl.
Apart from supporting the latest version of FFmpeg (at least when I started a couple of weeks ago), it's smaller, cleaner, and more complete than any previous version. Having said that this is essentially just a beta release.
This one is licensed GNU General Public License Version 3 (or later).
I've kinda had enough for the moment so it's a pretty bare home page, but it's there.
Merry XMAS!
One of the things I did was fill out/sync up the important enums - AVCodecID, AVPixelFormat, AVSampleFormat, and so on. Previously the pixel format and sample formats were also Java enums - which can be convenient at times and provides some more (albeit much much overvalued) 'type safety'.
This was fairly easy because the PixelFormat was a simple densely ordered C enum so i could map between the two with a simple +-1. Unfortunately someone decided to add a big hole in the middle of it sometime between 0.10 and 3.4, ... so I gave up and just converted it to a class holding static final int's, and to make it consistent I did that with the other enumerations as well. It doesn't really make the classes any harder to use and improves the class size and memory footprint. I just added some methods to access libav*'s metadata information so I can still map between string representations and so on.
I had to add a small compilation stage which extracts the enums from the header files and converts them to a C file which when compiled and run produces the Java source ... this seemed the absolute shortest path to ensuring I got accurate numbers based on the ffmpeg build configuration.
So after about a weeks worth of solid work it's grown somewhat (about 2KLOC Java, and 2KLOC C, counting lines with ";{}") and the TODO list is getting pretty short.
I would like to clean up the exception design a bit - unfortunately i'm just not very good at that (who is?) but i'd like to get better. The build system is clean and simple but could be improved and needs to include the aforementioned enum stuff, a dist target and versioning. Logging would be nice (both redirect ffmpeg to java.util.logging and some for jjmpeg itself). JJMediaWriter? Fix the license headers, add at least a README.
Not today though, today I drink.
So I guess the mood took me, I somehow ended poking away until the very late morning hours (4am) the last couple of nights hacking on jjmpeg. Just one more small problem to solve ... that never ended. Today I should've been working but i've given up and will write it off, it's nearly xmas break anyway so there's no rush, and i'm ahead of the curve anyway.
I got this ported over and playing video fairly easily, and then went through on a cleanup spree. I removed all the BufferedImage, multi-buffering, and scaling stuff and a few other experiments which never worked. Some api changes allowed me to consolidate more code into a base class, and some changes to AVStream necessitated a different approach to initialising the AVCodecContext (using AVCodecParameters). I made a few other little tweaks on the way.
The reason I removed the BufferedImage code is because I didn't want to pollute it with "platform specific" code. i.e. swing, javafx, etc. I've moved that functionality into a separate namespace (module?).
My first cut just took the BufferedImage code and put it into another class which provides the functionality by taking the current AVFrame from the JJMediaReader video stream. This'll probably do but when working on similar functionality for JavaFX I took a completely different approach - implementing a native PixelReader() so that the native code can decide the best way to write to the buffer. This is perhaps a little more work but is a lot cleaner to use.
jjmpeg1 lets you scale images 'directly' to/from primitive arrays or direct ByteBuffers in addition to AVFrame. Since they have no structure description (size, format), this either has to be passed in to the functions (messy) or stored in the object (also messy). jjmpeg1 used the latter option and for now I simply haven't implemented them.
The PixelReader mentioned above does implement it internally but for code re-use it might make sense to implement them with the structure information as explicit parameters, and use higher level objects such as PixelReader/Writer to track such information. On the other hand the native code has access to more information so it also makes sense to leave it there.
I went a bit further and created a re-usable super-class that does most of the work and toolkit specific routines only have to tweak the invocation. This approach hides libswscale behind another api. The slice conversions don't work properly but they're not necessary.
So far I had public constructors and `finalisers' because otherwise the reflection code failed. That's a bit too ugly (and `dangerous') so I made them private. The reflection code just had to look up the methods and set them Accessible.
Constructor cc = jtype.getDeclaredConstructor(Long.TYPE);
cc.setAccessible(true);
return cc.newInstance(p);
Whilst working on JJMediaReader I hit a snag with the issue of ownership. In most cases objects are either created anew and released (or gc'd) by the Java code, or are simply references to data managed elsewhere. I was addressing the latter problem by simply having an empty release() method for the instance, but that isn't flexible enough because some objects are created or referenced the the context determines which.
So I expanded the Java-side object tracking to include a `refer' method in addition to the 'resolve' method. `resolve' either creates a new instance or returns and existing one with a weak-reference object which will invoke the static release method when it gets finalised. `refer' on the other hand does the same thing but uses a different weak-reference object which does nothing.
I then noticed (the rather obvious) that if an object is created, it can't possibly 'go away' from the object tracking if it is still alive; therefore the `resolve' method was doing redundant work. So I created another `create' method which assumes the object is always a new one and simply adds it to the table. It can also do some checking but i'm pretty sure it can't fail ...
If on the other hand the underlying data was reference counted then the `resolve' method would be useful since it would be possible to lookup an existing object despite it being `released'. So i'll keep it in CObject.
As part of this change I also improved CObject in other ways.
I was storing the weak reference to the object itself inside the object so I could implement explicit release and to avoid copying the pointer. I removed that reference and only store the pointer now. The WeakReference it already tracked in a hash table so I just look it up if I need it. This lets me change the jni code to use a field lookup rather than a function call to retrieve it (I doubt it makes much perf difference but I will profile it at some point).
I also had some pretty messy "cross-layer" use of static variables and messy synchronisation code. I moved all map references to outside of the weak reference routine and use a synchronised map for the pointer to object table.
For explicit release I simply call .clear() and .enqueue() on the WeakReference - which seems to do the right thing, and simplifies the release code (at least conceptually) since it always runs on the same thread.
The task is to load a set of data from files, process the data, and collate the results. It's quite cpu intensive so is a good fit for parallelisation on modern cpus. Queuing theory would suggest the most efficient processing pipeline would be to run each processing task on it's own thread rather than trying to break the tasks up internally.
I tried a couple of different approaches:
Files.find().forEach() (serial to compare)
Files.find().parallel().collector(custom concurrent collector)
Files.find().parallel().flatMap().collect(toList())
The result was a bit pants. At best they utilised 2 whole cores and the total execution times were 1.0x, 0.77x, and 0.76x respectively of the serial case. The machine is some intel laptop with 4 HT cores (i.e. 8x threads).
I thought maybe it just wasn't throwing enough threads at it and stalling on the i/o, so I tried a separate flatMap() stage to just load the data.
Files.find().parallel().flatMap(load).flatMap(process).collect(toList())
But that made no difference and basically ran the same as the custom collector implementation.
So I hand-rolled a trivial multi-thread processing graph:
Files.find().forEach(load | queue)
queue | process | outqueue
outqueue | List.add()
Result was all 8x "cores" fully utilised and a running time 0.30x of the serial case.
I didn't record the numbers but I also had a different implementation that parallelised parts of the numerical calculation instead. Also using streams via IntStream.range().parallel() (depending on the problem size). Surprisingly this had much better CPU utilisation (5x cores?) and improved runtime. It's surprising because that is a much finer-grained concurrency with higher overheads and not applied to the full calculation.
I've delved into the stream implementation a bit trying to understand how to implement my own Spliterators and whatnot, and it's an extraordinarily large amount of code for these rather middling results.
Not that it isn't a difficult problem to solve in a general way; the stream "executor" doesn't know that I have tasks and i/o which are slow and with latency compared to many small cpu-bound tasks which it seems to be tuned for.
Still a bit disappointing.
Well for whatever reason I got stuck into redoing jjmpeg and seem to have written most of the code (90%?) after a couple of weekends. It was mostly mandraulic and a bit tedious but somehow surprisingly relaxing and engaging; a short stint of unchallenging work can be a nice change. A couple of features are still missing but the main core is done.
Unfortunately my hope that the ffmpeg api was more bindable didn't really pan out but it isn't really any worse either. Some of the nastiest stuff doesn't really need to be dealt with fortunately.
I transformed most of the getters and setters into a small number of simple macros, and thus that part is only about as much work as the previous implementation despite not needing a separate compilation stage. I split most of the objects into separate files to make them simpler to maintain and added some table-based initialisation helpers to reduce the source lines and code footprint.
It's pretty small - counting `;' there's only 750 lines of C and 471 lines of Java sources. The 0.x version has 800 lines of C and 900 lines of Java, a big portion of which is generated from an 800 line (rather unmaintainable) Perl script. And the biggest reduction is the compiled size, the jar shrank from 274KB to 73KB, with only a modest increase from 55KB to 71KB in the (stripped) shared library size (although the latter doesn't include the dvb or utility classes).
There's still a lot of work to do though, I still need to test anything actually works and port over the i/o classes and enum tables at the least, and a few more things probably. This is the boring stuff so it'll depend on my mood.
In other news I finally killed my PC - I tried one more time to play with the BIOS and after a few updates it got so unstable it just crashed during an update and bricked the motherboard. Blah. I discovered I could order a new BIOS rom so i've done that and i'll see if i can recover it, otherwise I might get another mobo if I can still get AM2+ boards here, or just get another machine. I'll probably look into the latter anyway as it's always been a bit of a hassle (despite working flawlessly when it does and it's a very nice small machine.
Well i've had reason to visit jjmpeg again for something and although it's still doing the job, it's a very very long way behind in version support (0.10.x?). I've added a couple of things here and there (recently AVFormatContext.open_input so I could open compressed webcam streams) but i'm not particularly interested in dropping another release.
But ... along the way I started looking into writing a new version that will be up to date with current ffmpeg. It's a pretty slow burner and i'm going to be pretty busy with something (relatively interesting, moderately related) for the next couple of months.
But regardless here are a few what-if's should I continue with the effort.
The old generator + garabage collection support required 4 classes per object.
Whilst most of it is autogenerated the generator sucks to maintain and it's a bit of a mess. I've also since learnt that cutting down the number of classes is desirable.
So instead i'll use the "CObject" mechanism with the WeakReference being a simple native pointer holder object which also knows how to free it. In this case at most 2 custom classes are required - one for autogenerated code (if that happens) and any helper/custom code.
A few things require reflection going this route but the overheads should be acceptable.
Native memory was wrapped in native ByteBuffer objects.
Originally the goal was to have java just access fields
directly but in practice this wasn't practical as the
structures change depending on the compile options so you end
up with both the C and Java code being system specific, and
the Java code requires a compiler to implement it (C being
handled by gcc). A side-goal was to make the Java library
bit-size independent without resoring to long -
although that's all ByteBuffer uses.
Because the objects are just wrapped on the pointer there is the possibility that multiple objects can be created to reference the same underlying C object (e.g. getStreams().get(0) repeated). Whilst this isn't as bad as it sounds one has to ensure the objects aren't holding any of their own state java-side for example. It also turns out that a direct ByteBuffer isn't terribly fast either from the Java side or looking up from the C side (not sure why on the latter).
CObject just uses a long directly, which also precludes the likelyhood of poking around C internals by accident or otherwise. It also ensures unqiue objects reference unique pointers - this requires some overhead but it isn't onerous.
Using two concrete classes per object allowed the internal details of passing pointers (ByteBuffer) around to be hidden from the luser.
CObject likewise uses an accessor to retrieve the native pointer, but because it's the super-class of all objects the objects can simply be passed in directly. That is native methods just look like java methods and so there is no need for any trampolines between method interfaces. It does require a bit more support on the JNI side when returning objects, but it's trivial code.
An alternative would be to use a long and pass that around but then you still need the public/native separation and all the hassle that entails.
A lot of the current binding is autogenerated.
Once the generator was written this was fairly easy to maintain but getting the generator complete enough was a lot of work. The biggest issue is that the api just isn't very consistent, and some just don't map very nicely to a Java api. Things such as out parameters - or worse, absolute snot like like AVDictionary that should never have existed (onya libav!).
Each case required special-case code in the generator, often extra support code, and sometimes a fall-back to manually writing the whole lot.
Working with zcl - and in that case the OpenCL apis are much cleaner and consistent - I discovered it was somewhat less work just to do it manually and not really any harder to maintain afterwards. At least in the case of the original ffmpeg the inconistency was simply because it wasn't originally intended as a public library, and I suspect the newer versions might be a bit better.
I'm still undecided about simple data accessors as a good case can be made for saving the typing if there are many to write. So perhaps they could still be autogenerated (to an abstract super class as now), or they could be parameterised like they are in zcl (i.e. internally getInt(field) with public wrappers). Another half and half option would be to use the C preprocessor to do most of the ugly work and still write the Java headers by hand. Probably the last one.
Does anyone else care?
disable_mtrr_cleanup did nothing. disable_mtrr_trim would hang the boot.
I noticed the RAM speed was wrong in the BIOS again so i reset it, and lo and behold it all showed up, but only until the next reboot. Back to bhe BIOS and just changed the ram from one speed to another and back again - RAM returns!
Erg.
Well I was up till 3am fucking around with this bloody machine.
After verifying the hardware actually works it seems that the whole problem with my RAM not being found is the damn BIOS. I downloaded a bunch of BIOSs intending to try an older one and realised I hadn't installed the latest anyway. So I dropped than in and low and behold the memory came back. Yay.
So now I had that, I thought i'd try and get OpenCL working again. Ok, installed the latest (17.40) amdgpu-pro ... and fuck. Unsupported something or other in the kernel module. Sigh. I discovered that 17.30 did apparently work ... but it took a bit of digging to find a download link for it as AMD doesn't link to older releases (at this point i also found out midori has utterly shithouse network code. sigh). I finally found the right one (that wasn't corrupt), installed it and finally ...
Oh, back to 3.5G ram again. FAAARK.
At this point the power went out for the whole street so I had a shower using my phone's torch and went to bed.
Did some more digging when I got up (I was going to give up and say fuck it, but i've come this far), tried manually adding the ram using memmap, and finally confirmed the problem was the BIOS. So i tried an older one. That worked.
But only for a while, and then that broke as well. So trying the previous one ... groan.
Mabye it's time to cut my losses, it's already 1pm, the sun is out, heading for a lovely 31 degrees.
I also got rocm installed but i don't know if it works on this hardware, although at least the kernel is running ok so far.
I didn't have much to do today and came across some articles about jfs and io schedulers and thought i'd run a few tests while polishing off a bottle of red (Noons Twleve Bells 2011). Actually i'd been waiting for a so-called "mate" to show up but he decided he was too busy to even let me know until after the day was over. Not like i've been suicidally depressed this week or anything.
The test I chose was to compile open jdk 9 which is a pretty i/o intensive and complex build.
My hardware is a Kaveri A10-7850K APU, "4G" of memory (sigh) on an ITX motherboard, and a Samsung SSD 840 EVO 250GB drive. For each test I blew away the build directory, re-ran configure, set the scheduler "echo foo > /sys/block/sda/queue/scheduler", and flushed the buffer caches "echo 3 > /proc/sys/vm/drop_caches" before running "time make images". I forced all cpu cores to a fixed 3.0Ghz using the performance scheduler.
At first I kept getting ICE's in the default compiler so I installed gcc-4.9 ... and got ICE's again. Inconsistently though, and we all know what ICE's in gcc means (it almost always means broken hardware, i.e. RAM). Sigh. It was afer a suspend-resume so that might've had something to do with it.
Well I rebooted and fudged around in the BIOS intending to play with the RAM clock but noticed I'd set the voltage too low. So what the hell I set it 1.60v and rebooted. And whattyaknow, suddenly I have 8G ram working again (for now), first time in a year. Fucking PCs.
Anyway so I ran the tests with each i/o scheduler on the filesystem and had no ICE's this time (which was dissapointing because it means the hardware isn't likely to be busted afterall, just temperamental). I used the default compiler. As it was a fresh reboot I ran builds from the same 80x24 xterm and the only other luser applications running beyond the panels and window manager were 1 other root exterm and an emacs to record the results.
cfq real 10m39.440s user 28m55.256s sys 2m40.140s deadline real 10m36.500s user 28m44.236s sys 2m40.788s noop real 10m42.683s user 28m43.960s sys 2m41.036s
As expected from the articles i'd read, deadline (not the default in ubuntu!) is the best. But it's hardly a big deal. Some articles also suggested using noop for SSD storage but that is "clearly" worse (oddly that it increases sys time if it doesn't do anything). I only ran one test each in the order shown so it's only an illustrative result but TBH I'm just not that interested especially if the resutls are so close anyway. If there was something more significant I might care.
I guess i'll switch to deadline anyway, why not.
No doubt much of the compilation ran from buffer cache - infact this is over 2x faster than when I compiled it on Slackware64 a few days ago - at the time I only had 4G system ram (less the framebuffer) and a few fat apps running. But I also had different bios settings (slower ram speed) and btrfs rather than jfs.
As an aside I remember in the early days of Evolution (2000-2-x) when it took 45 minutes for a complete build on a desktop tower, and the code wasn't even very big at the time. That dell piece of crap had woefully shitful i/o. Header files really kill C compilation performance though.
Still openjdk is pretty big, and i'm using a pretty woeful CPU here as well. Steamroller, lol. Where on earth are the ryzen apu's amd??
notzed@minized:~/hg/jdk9u$ find . -type f \ -a \( -name '*.java' -o -name '*.cpp' -o -name '*.c' \) \ | xargs grep \; | wc -l 3063124(some of the c and c++ is platform specific and not built but it's a good enough measure).
Well I found a way to make Midori usable for me as a browser-of-text using the user stylesheet thing.
Took some theme and stripped out the crap and came up with this ... it turns all the text readable but leaves most of the rest intact which is an improvement on how firefox rendered it's colour and stye overrides. It just overrode everything which broke a lot of style-sheet driven GUI toolkits amongst other sins.
* {
color: #000 !important;
text-shadow: 0 0 0px #000 !important;
box-shadow: none !important;
background-color: #777 !important;
border-color: #000 !important;
border-top-color: #000 !important;
border-bottom-color: #000 !important;
border-left-color: #000 !important;
border-right-color: #000 !important;
}
div, body {
background: transparent !important;
}
a, a * {
color: #002255 !important;
text-decoration: none !important;
}
a:hover, a:hover *, a:visited:hover, a:visited:hover *, span[onclick]:hover, div[onclick]:hover, [role="link"]:hover, [role="link"]:hover *, [role="button"]:hover *, [role="menuitem"]:hover, [role="menuitem"]:hover *, .link:hover, .link:hover * {
color: #005522 !important;
text-decoration: none !important;
}
a:visited, a:visited * {
color: #550022 !important;
}
I don't know if i'll stick with it yet but it's a contender, its built-in javascript blocker looks a lot better than some shitty plugin which will break every ``upgrade'' too. It doesn't seem to run javascript terribly fast, but that's the language's fault for being so shithouse.
This also supports `safe' explicit release which may be required in some circumstances where the gc is not run often enough.
It should work well with the JVM as it uses reference queues and no finalize methods. It requires minimal "extra" application support - just a class specific release() method.
I guess the main new thing is the module system. It probably has some warts but overall it looks quite decent. maven and `aficionados' of other modularisation systems seem to be upset about some things with it but I mean, maven?
I did a bit of playing with zcl to see how it could be modularised. At least on paper it's a very good fit for this project due to the native code used. In practice it seems a little clumsy, at least at my first attempt.
I decided to separate it into two parts - the main reusable library and the tools package. This required adding another top-level directory for each module (as the module name is used by the compiler), and a couple of simple module-info.java files.
au.notzed.zcl/module-info.java
module au.notzed.zcl { // module name
exports au.notzed.zcl; // package name
requires java.logging; // module name
}
au.notzed.zcl/tools/module-info.java
module au.notzed.zcl.tools { // module name
exports au.notzed.zcl.tools; // package name
requires au.notzed.zcl; // module name
}
With the source moved from src/au to au.notzed.zcl/au or au.notzed.zcl.tools/au as appropriate (sigh, yuck). Note that the name is enforced and must match the module name although the rest of the structure and where the module name exists in the path is quite flexible; here i obviously chose the simplest/shortest possible because I much prefer it that way.
The filenames in the Makefile were updated and a tiny change added is all that is needed to create both java modules at once. Makefile
zcl_JAVAC_FLAGS=-h build/include/zcl --module-source-path .
Yes I also moved to using java -h to create the jni header files as I noticed javah is now deprecated.
Ok that was easy. Now what?
The next part is to create a jmod file. This can be platform specific and include native libraries (and other resources - although i'm not sure how flexible that is).
The manual commands are fairly simple. After i've had a bit of play with it I will incorporate it into java.make with a new _jmods target mechanism.
build/jmods/zcl.jmod: build/zcl_built
-rm $@
mkdir -p build/jmods
jmod create \
--class-path build/zcl/au.notzed.zcl \
--libs jni/bin/gnu-amd64/lib \
--target-platform linux-amd64 \
--module-version $(zcl_VERSION) \
$@
As an aside it's nice to see them finally moving to gnu-style command switches.
Now this is where thing kind of get weird and I had a little misreading of the documentation at first (as a further aside I must say the documentation for jdk 9 is not up to it's normal standards at all, it doesn't even come with man pages). While a modularised jar can be used like any other at runtime whilst adding the benefits of encapsulation and dependency checking a jmod really only has one purpose - to create a custom JRE instance. As such I initially went the jmod route for both zcl and zcl.tools and found it was a bit clumsy to use (generating a whole jre for a test app? at least it was "only" 45MB!). The whole idea seems to somewhat fight against the 'write once run anywhere' aspect of java, even though i've had to create the same functionality separately for delivering desktop applications myself. For example it would be nice if jlink could be used to create a multi-platform distribution package for your components without including the jre as well (i.e. lib/linux-amd64, lib/windows-amd64 for native libraries and package up all the jars etc), but I guess that isn't the purpose of the tool and it will be useful for me nevertheless. There is definitely some merit to precise versioning of validated software (aka configuration management) although these days with regular security updates it spreads the task of keeping up to date a bit further out.
One nice thing is that the module path is actually a path of directories and not modules. No need to add every single jar file to the classpath, you just dump them in a directory. This wasn't possible with the classpath because the classpath was what defined the dependencies (albeit in a pretty loose way).
As I only cross-compile my software for toy platforms I was also curious how this was supposed to work ...
I found a question/answer on stack overflow that stated unequivocally that jlink was platform specific and that was that. This is incorrect. jlink is platform agnostic and you must supply the location of the system JDK jmods. The only problem is right now the only microsoft windows jdk available is an executable installer and no tar is available, one only hopes this is a temporary situation. So the binary must first be run inside some microsoft windows instance and then I believe I can just copy the files around. Maybe it will work in wine, but either way I haven't checked this yet.
One issue I do see as a free software developer is that once you jlink your modules, they are no longer editable (or cross platform). By design the modules are stored in an undocumented and platform-specific binary format. So here's the question ... how does this affect the GNU General Public License, and particularly the GNU Lesser General Public License? The former perhaps isn't much different from any statically linked binary -because GPL means all source must be GPL compatible. But in the case of the LGPL it is possible to link with non-GPL components - only in the case that the LGPL components may be replaced by the receiver of the software. In dynamic linking this can be achieved by ensuring all LGPL components are isolated in their own library and simply changing the load path or library file, but for static linking this requires that all object files are available for re-linking (hah lol, like anyone gives a shit, but that's the contract). So anyone distributing a binary will have to distribute all the modules that were used to build it together with the source of the LGPL modules. Yeah I can see that happening. Or perhaps the module path might be enough, and there is a mechanism for patching (albeit intended for development purposes).
Still, I think an article may be required from The Free Software Foundation to clarify the new java situation.
It would also be nice if jlink has support for bundling source which addresses much of the GPL distribution issue. Obviously it does because the jdk itself includes it's own source-code but it gets put into lib/ which seems an odd place to put it (again i don't know how flexible this structure is although it appears to be quite limited).
Update: Ahh covered by the classpath exception I guess, at least for the JRE. If one builds a custom JRE though any third party modules would also require the classpath exception? Or would including the source and all modules used to create the JRE suffice? Hmmm.
I would guess that the modularisation will have slow uptake because it's quite a big change and locks your code into java 9+, and it may evolve a little over the next jdk or two. I'm in two minds about using it myself just yet because of this reason and also because NetBeans has failed to deliver any support for it so far that i can tell (I was dissapointed to see Oracle abandon NetBeans to apache which is most probably part of it, and they're too busy changing license headers to get any real work done). There will also likely be blowback from those invested in existing systems, merits or not. And then there's dealing with the fuckup of a situation that android/"java" is in.
I myself will poke around with it for a while and merge the functionality into java.make, it's actually a pretty close fit to everything i've done (which is a nice validation that my solution wasn't far off) and will simplify it even if i might have to make a few minor changes like platform names.
Its a pretty good fit for my work but will require a bit of setup so I wont rush into it (a couple dozen lines of shell is doing a good enough job for me). Besides it's probably worth getting some experience with it before committing to a particular design. The NetBeans situation will also be a bit of a blocker and i'll probably wait for 9 to be released first.
A bunch of bugfixes, new build system, more robustness, and OpenCL 2.1 support.
There are still some thing i'm experimenting with - primarily the functional/task stuff as it's just not flexible enough - but it's stable and robust and easy to work with so i'm no longer using JOCL for anything at work.
On a personal note I still haven't really gotten back into hacking and i had a short sojourn into facebookland so i haven't had much to write about. It's mostly been work, very poor sleep, and drinking! Oh and I started wearing kilts ...
And so it goes.
Update: 29.11.16 And so it goes ...
