I somewhat embarassingly just discovered that FFmpeg already has some support for using libstagefright for hardware video decoding on Android.
I always thought this was the place it should go, since mucking around with OMX or other junk is just such a pain, and it belongs there.
Bit of a headfuck getting it to compile, particularly since this adds a C++ dependency, and then after all that ... it just crashes.
I/ffmpeg (10484): Assertion i < avci->buffer_count failed at /home/notzed/svn/jjmpeg-1.0/jjmpeg-core/jni/ffmpeg-1.0/libavcodec/utils.c:603 F/libc (10484): Fatal signal 11 (SIGSEGV) at 0xdeadbaad (code=1), thread 10607 (VideoDecoder) I/DEBUG ( 1894): *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** I/DEBUG ( 1894): Build fingerprint: 'samsung/m0zs/m0:4.1.2/JZO54K/I9300ZSEMB1:user/release-keys' I/DEBUG ( 1894): pid: 10484, tid: 10607, name: VideoDecoder >>> au.notzed.jjmpeg <<< I/DEBUG ( 1894): signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr deadbaad I/DEBUG ( 1894): r0 00000027 r1 deadbaad r2 40126b0c r3 00000000 I/DEBUG ( 1894): r4 00000000 r5 620f2a04 r6 60bfcc00 r7 61d0b8b0 I/DEBUG ( 1894): r8 5b127c80 r9 61bd5718 sl 000104d5 fp 61bd5718 I/DEBUG ( 1894): ip 5dc4bcbc sp 620f2a00 lr 400f8c65 pc 400f52fe cpsr 60000030 I/DEBUG ( 1894): d0 0000000200000000 d1 4b0000004b000021 I/DEBUG ( 1894): d2 000000094b000021 d3 0000000000000000 I/DEBUG ( 1894): d4 3ce5999061da5385 d5 3f34e1653f349a33 I/DEBUG ( 1894): d6 3f35287b3f356f75 d7 0106999ec0000000 I/DEBUG ( 1894): d8 0000000000000000 d9 0000000000000000 I/DEBUG ( 1894): d10 0000000000000000 d11 0000000000000000 I/DEBUG ( 1894): d12 0000000000000000 d13 0000000000000000 I/DEBUG ( 1894): d14 0000000000000000 d15 0000000000000000 I/DEBUG ( 1894): d16 8000000000000000 d17 ffffffffffffffff I/DEBUG ( 1894): d18 416347d4c0000000 d19 3fe0000000000000 I/DEBUG ( 1894): d20 3fe0000000000870 d21 0000000000000000 I/DEBUG ( 1894): d22 0000000000000000 d23 0000000000000000 I/DEBUG ( 1894): d24 0000000000000000 d25 0000000000000000 I/DEBUG ( 1894): d26 0000000000000000 d27 0000000000000000 I/DEBUG ( 1894): d28 3f3504f3bf3504f3 d29 bf3504f33f3504f3 I/DEBUG ( 1894): d30 0000000000000000 d31 3f3504f33f3504f3 I/DEBUG ( 1894): scr 20000010
So yeah, i dunno. Perhaps it is a bug in jjplayer, but I tried removing all output handling, and it still just crashes inside codec->decode(), and the api to that point is so simple I don't think I can screw it up.
I've wasted half a day on this and it's losing it's interest very fast ...
Update: So, just when I was about to give up, I found a threading bug in the codec implementation. This at least lets it decode more frames but it's still crashing later on inside the GLES library.
I haven't got the NV21 image format working properly yet, so it may be related to that, maybe.
Update: Well that was pretty much a full day down the drain, I submitted a patch to the ffmpeg-devel mailing list (yay, subscribe to a high volume list of zero interest to me just to submit a 10 line patch).
It turned out that the file I was testing against refuses to play at all in the bundled players, so it's something to do with the hardware/firmware. At best they load the first frame then abort, although they don't segfault (probably the parser/demux stops it going as far as the codec).
On a video recorded from the camera it seems to work ok, although it certainly doesn't appear particularly smooth - I would need to do some timings to see what's going on. I haven't written the NV21 support either. And the build stuff needs cleaning up and parameterising before I can check it in.
If it's going to still be so useless i'm not sure I really care to be honest. And I've had more than enough for today so whatever I decide will have to wait.