Well I finished moving the executive library over to use object interfaces, and managed to get enough of the bootstrap process done so that it can now launch a `user process' which may one day complete the job. Still no virtual memory management but it's a start.
Then I started looking at the process of creating and initialising libraries. InitResident is the function that creates libraries and devices from simple (rommable) definitions. It allocates the base object and sets up it's interfaces and may call init functions on either.
But the real issue is how to handle this type of shared library in a protected memory environment.
Basically the library (and device or resource) is just an object. It has a data area referenced through an instance pointer (with public and private parts), and it has a set of interfaces (function table). In the no-mmu case when you 'open' a library it (normally) only gets initialised once (for the entire system), and can contain global data within its `base' object. If it needs to do a lot of initialisation work (in time or space) then this can be a huge win since every process using the same library doesn't need to do it every time. But how to do that with multiple processes with virtual memory? In OS4 they just stick to using `public memory' - globally shared read/write memory. It's not as bad as it sounds, for a single-user system, but I'm looking at adding more protection.
Code is easy, it can just be relocated to the same virtual address range and then mapped to all processes. BSS+Data is a pita. I will need to either force no globals (no Data or BSS) or implement copy-on-write. Forcing no global data is fine for any libraries I write, but if I were to `wrap' existing libraries it might be too limiting. Library bases (the data component of library objects) are another issue. Basically once the init code is run, a library expects these to be initialised and accssible for all interface calls.
About the only idea I can see working is to completely re-init the library base again for every process that opens it. If the library wants some sort of global shared state it could start its own server process and use message ports to communicate; I will need to do this anyway for devices so maybe I'll put the feature into libraries instead. It loses some of the benefits but there are big gains in security and isolation.
The executive library is a special case, for one it needs to be set up before the setup functions inside it can be called. However it may make sense to have per-process data structures the library can access from process space too. So far I have had the supervisor-state code use the library base to store global system data, and I was thinking of having it accessible read-only to all processes. But maybe they can have their own copy of some of the data, e.g. memory allocator tables or current thread-id or thread-local-storage stuff. That way processes get their own copy of 'execbase', and the supervisor code has its own structure (or perhaps its own instance of the same structure to allow for maximal code re-use). Although I don't want to be duplicating any significant amount of data during a context switch a pointer or two might be ok. More thought required.