CSZ

Well I kept poking away at the XHTML/CSS stuff, for want of something better to do. I had a couple of wins along the way.

The cascading and inheritance is working somewhat better now, and I added a few more properties. Given that I'm not very familiar with all the various rules, I think I have a fairly efficient resolution mechanism by indexing various bits and pieces. The layout system is still crap, and very very incomplete, but at least I have baselines aligned now.

This is a totally contrived example, everything apart from the text layout and typefaces are hacked in one way or another.

I tried it on the kobo ... it's still fairly slow, but it's better (I think: TBH i can't remember what I tried the other stuff on). Still hampered by the text layout though.

For a 200k file (which is mostly just <p> elements), scanning the file, resolving the properties and generating the box list takes only a fraction of a second. I'm not trying to resolve or use very many properties though. About all i'm using `in anger' are some of the font-* properties.

But then performing the layout (as a single page) takes about 7s (once the jvm is warmed up), which is mostly due to TextLayout. I will have to try it with a simpler font than the one the JVM comes with.

Rendering is fairly ok (relative to the e-ink anyway) and all i'm doing is painting every textlayout in the whole tree ...

I uploaded it to MediaZ anyway, in the new CSZ module.

Hmmm, I should really take a break from hacking for a bit. But i'll believe that when I see it.

xhtml, css, boxes n shit

For some stupid reason I delved into CSS and XHTML and rendering thereof.

CSS is so deceptively simple: a few boxes, layouts in lines, and a few properties to set. Saying the devil is in the details here isn't doing the term justice. It's all in the details. And they're ugly.

From the fairly complex cascading rules, to the number of properties. The layout merging. The badly written documentation: filled with "x inline y box" "a block b box c" to such specificity, and no with definitions it is quite difficult to decipher what it's even talking about. It's also quite hard to debug, since it needs a fair bit of data structure to represent it.

CSZ

Anyway, after some mucking about, I have a relatively complete CSS lexer and parser, a fairly incomplete cascade resolver, a fairly incomplete layout engine, and a very incomplete style system. It's just enough to show paragraphs of text with some basic formatting. For a book reader I don't want the document to control the text too much anyway.

I'm attempting to do it while streaming the input, and (obviously thus) in a single pass using the pull parser from XMLStreamReader. Therefore initial parsing is quite quick, but it's still taking a relatively long time to lay out the boxes ...

TextLayout

And the problem here is TextLayout. It's just quite slow. I tried my own version of layout using FontMetrics.getCharsWidth(), but inside that just creates a TextLayout anyway, so it's even slower (or maybe not, now it's a bit faster?).

I know why it's so complicated; for laying out complex scripts and handling all the special cases. Anyway, that is the primary factor of constraint on performance at the moment, although as the implementation is so far from finished, i'm sure it wont be the last one.

(I played some more, and the font used plays a big part in the speed taken, so there's hope yet).

I suppose I should try it on the kobo to see how it goes there.

Dead end?

It's taken a lot of effort to get this far, and i'm not really happy with the result. So i'm not sure if i'll keep plugging away at it or throw it away (and if i need such a functionality, use cssbox). There's a lot left to get it to be useful for anything.

ReaderZ

I finally got around to checking in ReaderZ to mediaz.

I also tweaked a few things before I did:

• Added timer and timeout functionality to the GagdetZ event thread. Using a ScheduledExecutionService made this an absolute doddle, and infact the 'event thread' is now just the service.
  
  
• Changed the update mechanisms to try to speed up panning. It now pans using black and white updates and then does a full 'invert' refresh once you stop - after a 700ms timeout of inactivity. It is still a bit slow updating, but that might be because i'm rendering the content on the fly. I can't seem to get the e-ink to go any faster, although I did speed that up a bit over what it was.
  
  
• I took out the HTML backend for now. I'm using a very slightly modified build of CSSBox (to remove the logging dependencies) and just can't be bothered to work that all out. It's a bit slow anyway.
  
  
• For fun I added a picture viewer backend to the DocumentReader.
  
  
  

The README has all the other gory details.

I also had a good look at the text layout mechanism in Java ... boy what a nightmare of code that is. No wonder it's so slow. I tried to work out how it was doing it when it came down to it, but I couldn't fathom it before losing interest. The kobo html reader is about 10x faster at pagination; which just makes it all the more puzzling as to why the text reader is so slow.

Which has me thinking about C again, and using mupdf's stuff to format and render text instead ... well it would work for latin scripts anyway.

A browser ... ?

Ahh, so I totally didn't think I was going to even try to do this ...

But epub needs HTML, and I found CSSBox, and well one thing lead to another ...


So I've basically ported the SimpleBrowser example from CSSBox to ReaderZ - all it can do is render the page, and it lets you pan and zoom as with PDF files. No links or anything.

I tried rendering on the fly, and into an image - the former is a little slow to scroll (but not far off the e-ink speed), but I don't think it's clipping the drawing regions properly and so doing a lot more work than necessary anyway. Using an image scrolls fast but can't be zoomed very well, and takes more memory (I blew it with boycottnovell) ... so trade-offs.

It's pretty slow and clunky, but what can one expect when XML is involved?

Reader Shell

So I kept poking away at the browser code and my gadget toolkit.

It's getting fairly sophisticated now: I improved the StackLayout gadget to add filling and relative sizing glue. I added a list gadget - which works by pages, and is similar to JList, including a cell renderer, list model and selection model and I played around with a whole lot of other stuff as well.

So I have enough to finally create a reader shell: it presents a list of the files on the device, and lets you open them up with a pdf reader or a text reader, depending on the file type. Once inside it has a full-screen viewer with no visible buttons - but without buttons you can zoom, pan, change pages one at a time or flip through many a time. A popup menu (single short press in the middle of the screen) lets you quit back to the shell.

I cache the pagination for text files, so they open very quickly after the first visit, assuming the font settings haven't changed. The pagination descriptor is small, about 7k for a 500 page document. On a re-visit of the same file (i.e. once the jvm is warmed up), it's loading 500k text file in about 1/2 a second but even from a cold-start it's only about a second. PDF files also open fast, about the same speed. i.e. not much slower than the e-ink can refresh a single greyscale page. Closing a file and returning to the shell is similarly speedy.

And whilst the interface isn't very complicated, everything is still quite responsive, with no missed finger presses or long unexplained delays (although the first time you open a 500k text file, it still takes about 25s to re-paginate).

My panning is still a bit slow, although I am rendering the bitmap on the fly as well.

But ... i've pretty much done what I want for now: prove that the kobo touch e-reader is a zippy little unit, capable of much more performance than the included software lets it deliver.

I'll drop the code in MediaZ sometime in the next few days, and continue working on it for at least a while.

PDFZ

Checked in the first cut of PDFZ to the MediaZ project.

It's a simple binding to muPDF - i.e. a PDF renderer for Java that builds on arm and x64/x86 cpus.

Touch Interfaces 2

I was wondering how to get a decent user interface with only a finger ...

I came up with the idea of sensors (no doubt this is what everyone else already does, but I haven't used a touch device before let alone coded for one) that I can attach to a gadget, and the gadget can then decide what to do.

I have two types of sensors, drag sensors and press sensors. Drag sensors override any press sensors, and come into action once a drag has started. It only started after a certain threshold of movement is exceeded. Press sensors only activate if a press is detected without a drag. Additionally they can handle long (>300ms) or short presses separately.

So for a PDF viewer, I came up with the following 'soccer pitch' of sensors ...

• The semi-circle sensors are press sensors, and are used to pan around the page in screen-sized chunks in the obvious directions. I haven't implemented these yet.
  
• The left and right thirds of the entire screen are for page turning.
  
• The thin rectangle at the top of the screen is a drag sensor used for changing the zoom. There is also a coincident press sensor which is used to reset the zoom.
  
• The thin rectangle at the bottom of the screen is a drag sensor used for flipping through pages.
  
• The whole screen itself is a drag sensor, and used for finger panning. The whole screen also has a press sensor which will cause an e-ink 'cleanup' on a long press.

As far as this goes and even without implementing the panning buttons it makes a fairly comfortable PDF reader out of such a small screen. I might also need a way to go back to the top of the page but one-screen over, and back to the left of the page but one-screen down (and backwards of those?); although there is only so much space for such hidden buttons to be easy to use. I could either use the corners, or perhaps just do it based on exceeding the limit of the existing paging buttons.

PDFZ on Kobo

Well I got the PDF reader working on the kobo. The binding library 'just worked' thankfully, so I just cleaned the code up and worked on some performance issues.


Using a custom BufferedImage which talks directly to the ByteBuffer turned out to be very slow on the kobo jvm - about 10x slower than just copying the ByteBuffer to an RGB565 BufferedImage in a manual Java loop. So I just do that ... the images are only the size of the display so memory isn't a big issue.

It loads and renders a page in about 0.3s, so that is fast enough. I couldn't work out how to get it to 'fully' refresh automatically very cleanly so I just added a button to do it. Panning is a bit slow, even with monochrome update mode: OTOH finger events aren't lost when you press-slide-quickly-release.

For comparison here's the same text with the built-in viewer.


I think my last mention if it was a bit unkind - it isn't that bad, it turns out I had been reading a scanned paper, which tends to render poorly in everything. I found out that you can change pages whilst zoomed, although you either have to scroll off the side of the image first, or bring up the menu - and while the menu is up you get an icon of the 'zoom area' taking up a good chunk of the screen, so they're both a little bit clumsier than they might be. And some contrast enhancement wouldn't go astray when viewing text (although IMHO mupdf just renders text very well).

Incidentally, the dark square you can see in the lower left is really there: it's left over from an e-ink refresh after the zoom-preview-box is removed.

PDFZ, mupdf

Cut a long story short: i've come up with a simple binding to mupdf for Java.

I was going to just use jmupdf, but on closer inspection it had some issues I wasn't comfortable with. By writing my own it let me use the development version of mupdf: which has support for multiple threads and so on, and so removes the need for some of the work that had been done in jmupdf to 'libaryise' it. All I needed to do to mupdf was add -fPIC to CFLAGS so I could build a shared library out of the jni plugin.

Rather than bind the whole api (it's huge), i've bound a minimum required to load pdfs for display. A helper class written in C hides some of the mucking about when loading and rendering pages.

Obviously one thing that got me started on this was for a pdf viewer on the kobo touch ... although i'm not really convinced the display has enough resolution for it to work that well. I tried the built-in pdf viewer but it's pretty tedious to use - you can't turn pages unless you're fully zoomed out, and with the file I was testing it had the bounding box wrong so you had to do a lot of zoom in/out just to be able to read anything. Panning on the e-ink is a bit nasty too.

I haven't tried running it yet on the kobo: but i've already built the binaries using a cross-compilation environment so hopefully it should 'just work'. I've also made some progress on a GadgetZ based implementation, but it's super rough, here's a screenshot of it running on my workstation in my 'emulator'. I've upped the gamma somewhat in preparation for the e-ink and to test that functionality (which is why it is so dark).


I'm going to try to see if i can render the display list real-time, and only keep a frame-buffer sized backing for it. It's fast enough on the desktop, but i'm not sure about the ARM cpu on the kobo when it comes to panning (at the zoom setting shown in the picture it's taking about 0.002-0.004s to render on my workstation ... so it should be fast enough!).

The user interface with a single 'finger' input will be a challenge too ...

I crashed my kobo a couple of days ago whilst trying to work out suspend/resume, and spent a few wasted hours trying to activate it again after a factory reset since the windows pc I have access to suddenly decided to have it's hdd die ... so i'm a little reluctant to overuse it at the moment (although I ended up setting up the kobo application in virtualbox, should i need it again).

I've also had enough for today, so I'll look at the kobo another time.

Touch Interfaces

Although I should really be working on the shed or just getting some fresh air, I kept poking away at the kobo code.

I thought i'd try to implement a copy of the 'date setting' thing: the one on the kobo is really slow and hard to use for some reason and I wanted to verify that it wasn't a fault of the device ...


(this is similar-but-not the same as the kobo date setting page)

Rather than copy the same interface (which is based on selecting what to change and up/down buttons), I came up with a different one based on more 'standard' widgets: dropdown menu's and sliders.



I'm also using a new 'requester' class, which is a modal window which opens centered on the screen.

Of course, the problem with writing your own toolkit is you need to write all those things too ... but I managed to get those working eventually. The popup menu will try to align with the selected item, otherwise it fits to the screen.

I hit my first snag when trying to create requesters: the event loop iterates through a list of gadgets, but that ended up changing during the update since I was adding a new window ... so my first solution was to hack the popups in separately to the Screen class: it needs to track them for events anyway. I knew that although I wasn't hitting it at the moment, there was also a potential for multi-thread issues as well, so I attempted to create a work-around for that too. I removed access to the children of a Group gadget, and forced access through a synchronous interface which makes a copy of the list. Although it can potentially access stale data, it should eventually catch up.



I'm using a layout class for the layout of the menu content, so although it's not shown, the 'day' selector uses a grid layout and lists the day numbers in quasi-calendar format.



For the year I thought i'd try something different - a slider bubble, which just goes away once you lift your finger.



In action ... showing the faster update mode. I can also set it to update synchronously - in this case you get no artefacts, but it only updates about 3fps.

I also poked around with the update manager a fair bit. I had been just shipping repaint events to another thread and letting it merge them if it happened to have a few in the queue. But this meant the display would update quickly as soon as something changed and then usually the rest of the updates for handling that input would happen later: i.e. at least 2 updates (which are slow if you're waiting for them). So I now have it merge the updates as they come in, and only fire them off after processing input events. There is still some issues with this for the e-ink, e.g. if you do anything on an input 'pressed' event such as highlighting a button, and then perform the action on the 'released' event you still end up having to wait for at least one redundant updates - which again is slow if you're synchornising with the e-ink.

I also experimented with an 'invert' display every now and then: e.g. when a requester is closed. This is needed to clean up the e-ink fully. Here I hit some snags with synchronisation with the device driver although I have it sort of working. Sometimes it inverts the old data first, and then inverts the final data, even though i'm ensuring the updates are being sent in order. I am just using the same update maker though, so it probably has something to do with that.

Finally, I tried to work out how to resume from suspend but got nowhere - I can suspend the box but it wont resume on any button presses and I have to do a hardware reset.

GadgetZ ... on kobo.

So the projects of the ...Z's continue.

I played a bit with the thinlet toolkit - I managed to get some stuff to display, but I had trouble with the refreshing and mapping the input events properly. It also lacked some features and had a weird-arsed reflection based event system.

So, although I really didn't want to, I started from scratch with a 'simpler' toolkit. Which wasted all of today and half of yesterday ...



But after quite a bit of jiggery pokery I have some basic widgets working (as usual, it's the layout stuff that's a real shit to get working at all, let alone correctly). All events are handled in one thread, and all rendering his handled in another - so although you sometimes get artefacts, it ends up catching up: and the interface remains quite responsive in the meantime (so it should, there are at least 5 threads working away so nothing needs to block). The touch-screen does seem to have some issues with some areas of the screen but I don't think I can do anything about that.

I also worked out the input events for the two buttons (home and power) - for some reason when I tried it previously I got nothing. So when I get to that I can hook those up too.

The paint manager isn't terribly efficient - but it seems to be 'good enough' at this point even with piles of printf output. i.e. the cpu on the thing really is rather gutsy particularly compared to the slowness of the e-ink display.

Next time I play with it i might look at a text reader. Unfortunately paginating isn't quite as simple as i'd like - I wanted to read it off disk on the fly: but character set and word-wrap stuff makes this really messy. So to start with it might be based in memory.

Update Whilst I was writing this I had a bit more of a play. If I turn off anti-aliasing and use black and white for everything, the display updates much faster and with fewer artefacts.

Kobo Hacking

Update:See also the log of posts under the kobo label. I got quite far but still ended up stuck.

---

Hmm, so curiosity got the better of me and I played a bit more with my recently acquired kobo touch.

I downloaded the correct version of codesourcery gcc, and trivially compiled some binaries for it - being able to simply ftp code via wifi and run it in a telnet session provides for simple and rapid experimentation. I found also that if i just run the web browser and leave it on the google page the wifi stays active for longer; although I could always kill the ebook reader as well.

I ported a simple linux framebuffer example to run on it, and using the definitions and ioctls in mxcfb worked out how to tell the e-ink to refresh. It can be quite slow if you fully wait for the refresh, but that is only required sometimes by the looks of it. It seems to work a bit like an etch-a-sketch, and bits get left behind sometimes, requiring a full 'shake' once in a while to clean up the display (while reading, this makes the e-ink look even more like paper, since you seem to get the previous page 'showing through' as you do with most paperbacks).

I started playing with a simple 'hello world' using freetype to render some text ... but hell, that's just too much hassle. So I'm going to play a bit with Java first; Java on ARM is pretty gutless, so it may not be fast enough, but there's no harm in trying. The machine has 256MB/ram, so at least memory shouldn't be too much of a problem. It's a pity Java2D cannot write to a direct ByteBuffer so I will have to have a separate BufferedImage to render into, and some messy update code; but I think such overheads will be immeasurable compared to the e-ink update.

I tried the Sun JRE (actually it's about the only binary JRE i could find), which seems to work ok (at least with a simple 'hello world'), and now i'm working on a tiny bit of JNI to talk to the framebuffer display. Rather than have to do everything from scratch, I'm looking at porting a very simple toolkit 'thinlet' to work with this output device, and I'll also write some glue to the touchscreen input.

I've been really flat lately and so may not put much time into it, but the more I use the device the more the shitty software it comes with is pissing me off. It seems to be single-threaded and often blocked by inconsequential i/o operations - i.e. when it works the web browser is somewhat faster than the ebook reader is, so it's not so much the hardware or even the basic software, but some silly interactions going on in the background. Turning on aeroplane mode for example seems to eliminate many of the pauses.

e-reader

So ... I ended up cancelling my order with JB HI FI for the e-reader I ordered. For a retailer apparently struggling with on-line competition they really need to get their shit together: after 7 days, an 'in stock' item had still not shipped, let alone arrived. For online shops who have got their shit together, I typically have much larger items (e.g. a rotary clothes line!) arriving in only a few days.

So anyway, I ended up just going to officeworks; another evil corporation, but it's close. Originally when I looked last week they had it at RRP, then I was there Tuesday for something else and I saw it at the same price as JB Hi-fi, and by the time I decided to buy one anyway (I was nearly going to drop the whole idea ...) they'd dropped another 15$ off that price. Apart from having to stand around like an idiot for 10 minutes before anyone would serve me (unfortunately, the aren't just stacked on a shelf like everything else) it was an easy buy ...

Kobo Touch

So the device is a Kobo Touch.

The hardware is quite nice - solid feel. It's a bit smaller than I'd imagined - it's like a small paperback - but the screen is readable enough. A bit heavier than I expected too, but it's ok. I've never seen e-ink screens before, and the screen is better in brighter light but it's still ok in more subdued lighting. Screen updating is pretty slow - but for it's purpose to read text it is ok, it feels a it like very old LCD displays with a low battery. Using the 'sketch' tool or an event reader from a login shell, the touch input isn't perfect but does seem fine enough and quite responsive.

But the software ... is pretty crap. Usually it's ok, but sometimes it's super-slow. Dunno why - the hardware is beefy enough. Text files (book sized) are really slow, as it seems to re-paginate the whole thing every time you open a book, and for such text files it always forgets where you were up to. Changing display preferences on such a text file also takes about 30 seconds per change ... The touch input (apart from the sketch tool) is a bit hit and miss some of the time, and it is often un-clear if the tool is just too busy to process your input, or it just didn't pick it up in the first place. Which leads to double-actions or nothing happening.

I managed to get telnet working and installed strace (from the opensuse arm7l rpm) to see what the GUI was doing. During one of the long pauses where top showed the system wasn't very busy it spent an inordinate amount of time trying to read a config file with the following sequence was being executed thousands of times:

stat64("/mnt/onboard/.kobo/Kobo/Kobo eReader.conf", {st_mode=S_IFREG|0755, st_size=3495, ...}) = 0
stat64("/mnt/onboard/.kobo/Kobo/Kobo eReader.conf", {st_mode=S_IFREG|0755, st_size=3495, ...}) = 0
stat64("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2202, ...}) = 0
stat64("/mnt/onboard/.kobo/Kobo.conf", 0x7eacbfc8) = -1 ENOENT (No such file or directory)
stat64("/mnt/onboard/.kobo/Kobo.conf", 0x88f080) = -1 ENOENT (No such file or directory)
stat64("/mnt/onboard/Kobo/Kobo eReader.conf", 0x7eacbfc8) = -1 ENOENT (No such file or directory)
stat64("/mnt/onboard/Kobo/Kobo eReader.conf", 0x88f228) = -1 ENOENT (No such file or directory)
stat64("/mnt/onboard/Kobo.conf", 0x7eacbfc8) = -1 ENOENT (No such file or directory)
stat64("/mnt/onboard/Kobo.conf", 0x88f228) = -1 ENOENT (No such file or directory)

Which looks like pretty sloppy coding/mis-using a tool-kit.

The reader application turns off the wifi when you restart it, so I gave up for now rather than trying to fight with it to get stuff done. Might read some books first ...

When I get an arm compiler going again (my beagleboard stuff is all backed up ... somewhere), I might have a closer look. Installing new binaries is pretty easy, and there's some info on the forums about the touch screen and frame-buffer.

Another year.

Well i've been pretty lazy since my leave started - well not entirely lazy just not wedded to the computer keyboard as I usually am. Not really missing it either, yet.

I did however install an automatic watering system and with looking after the garden that keeps me busy enough. The main tomato plant is nearly spent (after about 13kg of tomatoes, not bad for a 2 year-old plant), but i've had a pile of purple beans and the fresh sweet-corn is very nice too and all the citrus is booming now the wine barrels aren't drying out every warmish day.

I went to the country for xmas with my sister & niece, my twin brother and mum; but after a week of that and them it was a relief to come home. I was so over drinking by then I went to bed early on NYE and have been dry since - it didn't help that I had hardly any sleep for weeks beforehand and during so my mood was all over the place. I couldn't keep up with her nightly drinking and she's too much of a know-it-all to put up with for too long!

After returning we had a bit of a heat-wave, so I just sat inside with the AC on, sleeping a lot and watching a lot of shit on tv. I'm a bit full of that too so I've started trying to do some recreational reading again - it's been a long long time - a few years - since I read anything. Starting with reading EE Doc Smith's Lensman series: the 40s style romance and kill-everyone action is a bit funny but it's better than repeats of some shitty american sitcom ... On an impulse buy I even ordered an 'ebook reader', although to my chargrin an 'in sock' item still hasn't shipped 4 days later ... reading on the laptop is ok but I'm interested to see how purpose-built device will compare.

Given I was off the booze for a few days I dropped the coffee too (until this morning) to see if that was the cause of my sleep apnoea going off the scale lately: unfortunately it had no noticeable effect. I was still tired and able to nap for hours every afternoon (thankfully the work next-door has ceased for the moment which let me `sleep'). So it's most probably just being too lazy and sedentary, too overweight, and perhaps some hay fever.