Binary Protocol

So I mowed the lawn and swept the yard and did some of the things i've been neglecting for a few weeks. Although now with no milk in my coffee I wasn't in need of the shop ...

By late afternoon i was a bit bored so I had a look at the binary protocol work I had started.

I decided to change tack rather than use the manual serialisation I had started. It was a variant of Externalizable, which although the fastest mechanism requires the client and server to have total knowledge of each end. Being java there are other possibilities such as sending the decoding library to the client at run-time, but I wanted to keep things simple.

I started creating general purpose containers because I saw I was just re-typing the same stuff again, and there was a fair bit of re-use possible. After a few iterations I refined it and came up with something i'm pretty happy with. Basically it's a typed and tagged data structure which can be encoded in such a way that it can only be decoded one way. It is a bit like BER encoding but stripped to the bare-bones without any of the complication only a committee and the requirement to solve every possible conceivable future use can provide. But I can also include custom objects for efficiency or convenience sake.

So this defines the syntax - defining, encoding, decoding, and access.

But the semantics are completely separate and are defined by the application. This is very important as for example it allows me to create messages with variable content or new messages that the client/server can ignore if they wish - without breaking the protocol exchange.

DuskMessage

So I basically have a thing called a dusk message, and every dusk message has a type and a name. The type defines the container, and the name is application specific. They are encoded on the wire as 2 separate bytes, limiting each to 256 which is more than enough for an application-specific protocol.

DuskMessage {
    byte type;
    byte name;
};

This on it's own is enough to pass simple notification messages that require no arguments. Then there are the simple types, which includes byte, short, int, long, float, and string.

ValueMessage<T> {
    DuskMessage;
    T value;
};

And I created a couple of list types, one is just a list of strings, and the other is a list of DuskMessages (i.e. 'any's) which is how more complex datatypes are created without requiring custom types.

ListMessage {
    DuskMessage;
    List<DuskMessage> value;
};

(etc, you get the idea)

List are just encoded in the obvious way - a length (short), followed by length-items of the DuskMessage. I've got some simple accessors like getByte(int name) and so on for easy client use.

There are also some custom types such as the map and entity updates which occur very frequently which should benefit from the reduced protocol and run-time overhead. I just hard-coded these into the system but with a small amount of work it would be possible to turn it into a reusable and extensible library where customisations were provided by the application code.

And finally I created another copy of all of these which also contain an ID - for messages which are intended for a specific Entity. They don't occur very often but it provides a bit more flexibility and efficiency for a small coding overhead.

Of course when I went to use it there were bugs in my code so I then wrote some protocol dumping routines which dump to a human-readable format. As a human never needs to read it other than for debugging there isn't much point sending a human-readable format for any other reason.

DuskProtocol

The semantics are then defined by the DuskProtocol interface. This is just a pile of integer constants which define the names to use for objects. Any class in the client or server that wants to use the constants just 'implements' the interface to gain access to them. There are two sets of constants.

MSG_*

The message constants define the message type and are set on the name of the highest level message. For simple types this is all they contain but the list can contain others. I decided to make these globally unique, although they could also be made unique-per-container-type. I tried that at first but it just got too messy and complicated to manage to make it worth doing. The more I worked on this the simpler it got, which is always a nice side-effect.

FIELD_*

The field constants define fields in compound (ListMessage) messages. These only have to be unique to each message type. I initially tried to work on creating a global namespace for them where I could pick and choose what to include but that just complicated things again and all it did was save some source-code space and one-off typing. I will move to per-message or at least related-groups grouping, and start each group at 0. Otherwise it's just unmaintainable.

So for example the player info update which is the most complex message (all the str/dex and so on) has some constants like:

    public final static int MSG_INFO_PLAYER = 15;
  ...
    public final static int FIELD_INFO_CASH = 0;
    public final static int FIELD_INFO_EXP = 1;
    public final static int FIELD_INFO_STR = 2;
    public final static int FIELD_INFO_STRBON = 3;
  ...

Currently i'm sending all of them every time - as the previous version did - but I can cut out things which aren't changes now, without having to change the client code (if i implement it the right way).

I did have a thought of having the server send some meta-data in an initial protocol exchange (say, allowing symbolic names for all messages and fields), but what the fuck for? It's already engineered enough to work, it doesn't need more than that.

So why bother/why not use another library/etc?

Firstly i find this kind of problem solving fun - it's just a never-ending puzzle trying to aim for a good solution knowing there are always trade-offs.. I've also done it quite a few times in the past - from LDAP which uses BER, to object serialisation in Evolution/camel, to using BerkelyDB, and plenty of other places besides. It just keeps cropping up and although I know this wont solve every problem it works for this one.

What I don't find particularly fun is learning how some behemoth general-purpose library works in the first place, let alone learning it's bugs and limitations. Even something as "trivial" as json requires some rather large library, and with all that it doesn't even interoperate reliably without fucking around.

For the application itself, as mentioned having a protocol with atomically-decodable messages allows for extension without completely breaking the protocol.

Having dynamic messages means I don't have to write code to handle dynamic messages into a custom protocol; it's already provided. Field access requires code for each field, a general list can be iterated - potentially reducing code required too. All the parsing and i/o handling is done and all I have to do is use the objects or their accessors.

It also means the encode/decode is in one place, and not scattered throughout the code as printf statements. So I can also force encapsulation at each end by forcing communications through this choke point. Which provides all sorts of benefits ...

... such as if i should decide to change the wire protocol in the future, or more likelty to support "web friendly" text-mode protocols.

In DuskZ

So I didn't just design the protocol syntax last night (ok it stretched into the night a bit - nothing on tv), I implemented it and changed DuskServer and DuskZ to use it. And debugged it enough to the point that you can log in and play some of the game ...

At this point the client is still talking the original protocol outward (i.e. a command line!), but I will need to change that to support some required features. For example scripts are able to ask the user questions but at the moment do some really nasty hackery like telling the client to pause, then ask the question, then flush the incoming stream, then hope it got it right. That can all go.

And the server protocol still has the same basic structure with only some minor tweaks.

There are other messages which could be combined in a more general 'update some object' type message. e.g. set range could become an optional field in an 'update player' message (although it looks like range isn't sent anyway). And others that could be combined for other reasons, such as wanting an atomic change of state, such as entering a battle - rather than sending a bunch of messages for each individual entity. Fortunately with the simple design I can just as easily embed messages inside a list as a field.

I also had a think about the auth protocol - right now it just creates a new user if you didn't exist, but it might be useful to get the player to confirm their password, and also go through the "new user" stuff like choosing a race all in one go.

Before i commit I want to clean up some of the naming conventions (netbeans refactor tools have had a work-out this week, although they're not bug free they work surprisingly well with 'broken' code), see about moving some of the fields to grouped messages, and change the login stuff. The login stuff goes a bit deeper so will require more work.

Update: So I managed to do most of what I intended - boy it was a lot more work than I thought. Getting the login working at both ends took a bit more thinking than I expected, but I managed to fit in a query mechanism that lets the game query the user for arbitrary values (currently: an item from a list) and the same convention is used during user creation to ask for whatever is needed. It took a while to work out how the current code needed it all to work too, and then I merged some of the message types (e.g. resize map + update images became map init), and then had to fix the client. It isn't quite done but it's enough to login and play around a bit.

Here's a protocol dump of the client creating a new user. It still goes through the same single login window I did before, but the client behaviour can be altered quite freely without changing the protocol. e.g. it could instead show 'unknown user', with a button for 'create' which runs a setup wizard locally, and then creates the user at the end.

sending: MSG_AUTH ListMessage name=0 value = {
        StringMessage name=3 value=x username
        StringMessage name=4 value=x password
}
state=Username: MSG_AUTH ListMessage name=0 value = {
        ListMessage name=2 value = {
                ListMessage name=0 value = { name = name of response string below
                        StringMessage name=0 value=Choose race
                        StringListMessage name=1 value= {
                          'lizardman'
                          'ork'
                          'indian'
                          'demon'
                          'elf'
                          'darkelf'
                          'halfling'
                          'dwarf'
                          'human'
                        }
                }
        }
        EntityIntegerMessage id=-1 name=0 value=3
        StringMessage name=1 value=Insufficient information to create player.
}
sending: MSG_AUTH ListMessage name=0 value = {
        StringMessage name=3 value=x
        StringMessage name=4 value=x
        ListMessage name=2 value = {
                StringMessage name=0 value=human name = name of choice above
        }
}
state=Username: MSG_AUTH ListMessage name=0 value = {
        EntityIntegerMessage id=195 name=0 value=0
        StringMessage name=1 value=Login ok.
}
state=Ready: MSG_CHAT StringMessage name=7 value=DuskZ Server 3.0 dev ...

So fairly clean and concise, and no screen-scraping required. And the query can include as many questions as needed, ... or even more complex content with some appropriate glue (such as a web form using the WebView?).