When we first started developing Mirage in 2009, we were rewriting huge chunks of operating system and runtime code in OCaml. This ranged from low-level device drivers to higher-level networking protocols such as TCP/IP or HTTP. The changes weren't just straight rewrites of C code either, but also involved experimenting with interfaces such as iteratees and lightweight threading to take advantage of OCaml's static type system. To make all of this easy to work with, we decided to lump everything into a single Git repository that would bootstrap the entire system with a single make invocation.

Nowadays though, Mirage is self-hosting, the interfaces are settling down, the number of libraries are growing every day, and portions of it are being used in the Xen Cloud Platform. So for the first developer release, we wanted to split up the monolithic repository into more manageable chunks, but still make it as easy as possible for the average OCaml developer to try out Mirage.

Thanks to much hard work from Thomas and his colleagues at OCamlPro, we now have OPAM: a fully-fledged package manager for Mirage! OPAM is a source-based package manager that supports a growing number of community OCaml libraries. More importantly for Mirage, it can also switch between multiple compiler installations, and so support cross-compiled runtimes and modified standard libraries.

OPAM includes compiler variants for Mirage-friendly environments for Xen and the UNIX tuntap backends. The installation instructions now give you instructions on how to use OPAM, and the old monolithic repository is considered deprecated. We're still working on full documentation for the first beta release, but all the repositories are on the Mirage organisation on Github, with some of the important ones being:

• mirage-platform has the core runtime for Xen and UNIX, implemented as the OS module.

• mirage-net has the TCP/IP networking stack.

• ocaml-cstruct has the camlp4 extension to manipulate memory like C structs, but with type-safe accessors in OCaml.

• ocaml-xenstore has a portable implementation of the Xenstore protocol to communicate with the Xen management stack from a VM (or even act as a server in a stub domain).

• ocaml-dns is a pure OCaml implementation of the DNS protocol, including a server and stub resolver.

• ocaml-re is a pure OCaml version of several regular expression engines, including Perl compatibility.

• ocaml-uri handles parsing the surprisingly complex URI strings.

• ocaml-cohttp is a portable HTTP parser, with backends for Mirage, Lwt and Core/Async. This is a good example of how to factor out OS-specific concerns using the OCaml type system (and I plan to blog more about this soon).

• ocaml-cow is a set of syntax extensions for JSON, CSS, XML and XHTML, which are explained here, and used by this site.

• ocaml-dyntype uses camlp4 to generate dynamic types and values from OCaml type declarations.

• ocaml-orm auto-generates SQL scheme from OCaml types via Dyntype, and currently supports SQLite.

• ocaml-openflow implements an OCaml switch and controller for the Openflow protocol.

There are quite a few more that are still being hacked for release by the team, but we're getting there very fast now. We also have the Mirage ports of SSH to integrate before the first release this year, and Haris has got some interesting DNSSEC code! If you want to get involved, join the mailing list or IRC channel!

On all hosts running Xen, there is a critical service called xenstore. Xenstore is used to allow untrusted user VMs to communicate with trusted system VMs, so that

• virtual disk and network connections can be established

• performance statistics and OS version information can be shared

• VMs can be remotely power-cycled, suspended, resumed, snapshotted and migrated.

If the xenstore service fails then at best the host cannot be controlled (i.e. no VM start or shutdown) and at worst VM isolation is compromised since an untrusted VM will be able to gain unauthorised access to disks or networks. This blog post examines how to disaggregate xenstore from the monolithic domain 0, and run it as an independent stub domain.

Recently in the Xen community, Daniel De Graaf and Alex Zeffertt have added support for xenstore stub domains where the xenstore service is run directly as an OS kernel in its own isolated VM. In the world of Xen, a running VM is a "domain" and a "stub" implies a single-purpose OS image rather than a general-purpose machine. Previously if something bad happened in "domain 0" (the privileged general-purpose OS where xenstore traditionally runs) such as an out-of-memory event or a performance problem, then the critical xenstore process might become unusable or fail altogether. Instead if xenstore is run as a "stub domain" then it is immune to such problems in domain 0. In fact, it will even allow us to reboot domain 0 in future (along with all other privileged domains) without incurring any VM downtime during the reset!

The new code in xen-unstable.hg lays the necessary groundwork (Xen and domain 0 kernel changes) and ports the original C xenstored to run as a stub domain.

Meanwhile, thanks to Vincent Hanquez and Thomas Gazagnaire, we also have an OCaml implementation of xenstore which, as well as the offering memory-safety, also supports a high-performance transaction engine, necessary for surviving a stressful "VM bootstorm" event on a large server in the cloud. Vincent and Thomas' code is Linux/POSIX only.

Ideally we would have the best of both worlds:

• a fast, memory-safe xenstored written in OCaml,

• running directly as a Xen stub domain i.e. as a specialised kernel image without Linux or POSIX

We can now do both, using Mirage! If you're saying, "that sounds great! How do I do that?" then read on...

Step 1: remove dependency on POSIX/Linux

If you read through the existing OCaml xenstored code, it becomes obvious that the main uses of POSIX APIs are for communication with clients, both Unix sockets and for a special Xen inter-domain shared memory interface. It was a fairly painless process to extract the required socket-like IO signature and turn the bulk of the server into a functor. The IO signature ended up looking approximately like:

type t
val read: t -> string -> int -> int -> int Lwt.t
val write: t -> string -> int -> int -> unit Lwt.t
val destroy: t -> unit Lwt.t

For now the dependency on Lwt is explicit but in future I'll probably make it more abstract so we can use Core Async too.

Step 2: add a Mirage Xen IO implementation

In a stub-domain all communication with other domains is via shared memory pages and "event channels". Mirage already contains extensive support for using these primitives, and uses them to create fast network and block virtual device drivers. To extend the code to cover the Xenstore stub domain case, only a few tweaks were needed to add the "server" side of a xenstore ring communication, in addition to the "client" side which was already present.

In Xen, domains share memory by a system of explicit "grants", where a client (called "frontend") tells the hypervisor to allow a server (called "backend") access to specific memory pages. Mirage already had code to create such grants, all that was missing was a few simple functions to receive grants from other domains.

These changes are all in the current mirage-platform tree.

Step 3: add a Mirage Xen "main" module and Makefile

The Mirage "main" module necessary for a stub domain looks pretty similar to the normal Unix userspace case except that it:

• arranges to log messages via the VM console (rather than a file or the network, since a disk or network device cannot be created without a working xenstore, and it's important not to introduce a bootstrap problem here)

• instantiates the server functor with the shared memory inter-domain IO module.

The Makefile looks like a regular Makefile, invoking ocamlbuild. The whole lot is built with OASIS with a small extension added by Anil to set a few options required for building Xen kernels rather than regular binaries.

... and it all works!

The code is in two separate repositories:

• ocaml-xenstore: contains all the generic stuff

• ocaml-xenstore-xen: contains the unix userspace and xen stub domain IO modules and "main" functions

• (optional) To regenerate the OASIS file, grab the add-xen branch from this OASIS fork.

Example build instructions

If you want to try building it yourself, try the following on a modern 64-bit OS. I've tested these instructions on a fresh install of Debian Wheezy.

First install OCaml and the usual build tools:

apt-get install ocaml build-essential git curl rsync

Then install the OCamlPro opam package manager to simplify the installation of extra packages

git clone git://github.com/OCamlPro/opam.git
cd opam
make
make install
cd ..

Initialise OPAM with the default packages:

opam --yes init
eval `opam config -env`

Add the "mirage" development package source (this step will not be needed once the package definitions are upstreamed)

opam remote -add dev git://github.com/mirage/opam-repo-dev

Switch to the special "mirage" version of the OCaml compiler

opam --yes switch -install 3.12.1+mirage-xen
opam --yes switch 3.12.1+mirage-xen
eval `opam config -env`

Install the generic Xenstore protocol libraries

opam --yes install xenstore

Install the Mirage development libraries

opam --yes install mirage

If this fails with "+ runtime/dietlibc/lib/atof.c:1: sorry, unimplemented: 64-bit mode not compiled in" it means you need a 64-bit build environment. Next, clone the xen stubdom tree

git clone git://github.com/djs55/ocaml-xenstore-xen

Build the Xen stubdom

cd ocaml-xenstore-xen
make

The binary now lives in xen/_build/src/server_xen.xen

Deploying on a Xen system

Running a stub Xenstored is a little tricky because it depends on the latest and greatest Xen and Linux PVops kernel. In the future it'll become much easier (and probably the default) but for now you need the following:

• xen-4.2 with XSM (Xen Security Modules) turned on

• A XSM/FLASK policy which allows the stubdom to call the "domctl getdomaininfo". For the moment it's safe to skip this step with the caveat that xenstored will leak connections when domains die.

• a Xen-4.2-compatible toolstack (either the bundled xl/libxl or xapi with some patches)

• Linux-3.5 PVops domain 0 kernel

• the domain builder binary init-xenstore-domain from xen-4.2/tools/xenstore.

To turn the stub xenstored on, you need to edit whichever init.d script is currently starting xenstore and modify it to call

init-xenstore-domain /path/to/server_xen.xen 256 flask_label

Something we've been working on for a little while now that we're pretty excited about is an OpenFlow implementation for Mirage. For those who're not networking types, in short, OpenFlow is a protocol and framework for devolving network control to software running on platforms other than the network elements themselves. It consists of three main parts:

• a controller, responsible for exercising control over the network;

• switches, consisting of switching hardware, with flow tables that apply forwarding behaviours to matching packets; and

• the protocol, by which controllers and switches communicate.

For more -- and far clearer! -- explanations, see any of the many online OpenFlow resources such as OpenFlowHub.

Within Mirage we have an OpenFlow implementation in two parts: individual libraries that provide controller and switch functionality. Linking the switch library enables your application to become a software-based OpenFlow switch. Linking in the controller library enables your application to exercise direct control over OpenFlow network elements.

The controller is modelled after the NOX open-source controller and currently provides only relatively low-level access to the OpenFlow primitives: a very cool thing to build on top of it would be a higher-level abstraction such as that provided by Nettle or Frenetic.

The switch is primarily intended as an experimental platform -- it is hopefully easier to extend than some of the existing software switches while still being sufficiently high performance to be interesting!

By way of a sample of how it fits together, here's a skeleton for a simple controller application:

type mac_switch = {
  addr: OP.eaddr;
  switch: OP.datapath_id;
}

type switch_state = {
  mutable mac_cache:
        (mac_switch, OP.Port.t) Hashtbl.t;
  mutable dpid: OP.datapath_id list
}

let switch_data = {
  mac_cache = Hashtbl.create 7;
  dpid = [];
}

let join_cb controller dpid evt =
  let dp = match evt with
      | OE.Datapath_join c -> c
      | _ -> invalid_arg "bogus datapath_join"
  in
  switch_data.dpid <- switch_data.dpid @ [dp]

let packet_in_cb controller dpid evt =
  (* algorithm details omitted for space *)

let init ctrl =
  OC.register_cb ctrl OE.DATAPATH_JOIN join_cb;
  OC.register_cb ctrl OE.PACKET_IN packet_in_cb

let main () =
  Net.Manager.create (fun mgr interface id ->
    let port = 6633 in
    OC.listen mgr (None, port) init
  )

We've written up some of the gory details of the design, implementation and performance in a short paper to the ICC Software Defined Networking workshop. Thanks to some sterling work by Haris and Balraj, the headline numbers are pretty good though: the unoptimised Mirage controller implementation is only 30--40% lower performance than the highly optimised NOX destiny-fast branch, which drops most of the programmability and flexibility of NOX; but is about six times higher performance than the fully flexible current NOX release. The switch's performance running as a domU virtual machine is indistinguishable from the current Open vSwitch release.

For more details see the paper or contact Mort, Haris or Anil. Please do get in touch if you've any comments or questions, or you do anything interesting with it!

The team signed up to do a tutorial at CUFP on the topic of Building a Functional OS, which meant zooming off to Tokyo! This was the first public show of the project, and resulted in a furious flurry of commits from the whole team to get it ready. The 45-strong crowd at the tutorial were really full of feedback, and particular thanks to Michael for organising the event, and Yaron, Marius, Steve, Wil, Adrian and the rest for shouting out questions regularly!

• The tutorial is a Mirage application, so you can clone it and view it locally through your web browser. The content is mirrored at tutorial.openmirage.org, although it does require cleanup to make it suitable to an online audience. The SVG integration is awkward and it only works on Chrome/Safari, so I will probably rewrite it using deck.js soon. The tutorial is a good showcase of Mirage, as it compiles to Xen, UNIX (both kernel sockets and direct tuntap) with a RAMdisk or external filesystem, and is a good way to mess around with application synthesis (look at the Makefile targets in slides/).

• Installation: instructions have been simplified, and we now only require OCaml on the host and include everything else in-tree. Thomas has also made Emacs and Vim plugins that are compatible with the ocamlbuild layout.

• Lwt: a new tutorial which walks you through the cooperative threading library we use, along with exercises (all available in mirage-tutorial). Raphael and Balraj are looking for feedback on this, so get in touch!

• Javascript: via node.js did not work in time for the tutorial, as integrating I/O is a tangled web that will take some time to sort out. Raphael is working on this in a separate tree for now. As part of this effort though, he integrated a pure OCaml regular expression library that does not require C bindings, and is surprisingly fast.

• Devices: we can now synthesise binaries that share common code but have very different I/O interfaces. This is due to a new device manager, and David also heroically wrote a complete FAT12/16/32 library that we demonstrated. Yaron Minsky suggested a different approach to the device manager using first-class modules instead of objects, so I am experimentally trying this before writing documentation on it.

• TCP: the notorious Mirage stack is far more robust due to our resident networking guru Balraj hunting down last-minute bugs. Although it held together with sticky tape during the tutorial, he is now adding retransmission and congestion control to make it actually standards-compliant. Still, if you dont have any packet loss, the microkernel version of this website does actually serve pages.

• OpenFlow: is a new standard for Software Defined Networking, and Haris and Mort have been hacking away at a complete implementation directly in Mirage! We will be giving a tutorial on this at the OFELIA summer school in November (it is summer somewhere, I guess). The prospect of a high-speed microkernel switching fabric for the cloud, programmed in a functional style, is something I am really looking forward to seeing!

• Jane Street Core: preceeding us was Yaron's Core tutorial. Since Mirage provides it own complete standard library, we can adopt portions of Core that do not require OS threads or UNIX-specific features. I really like the idea that Mirage enforces a discipline on writing portable interfaces, as dependencies on OS-specific features do sneak in insiduously and make switching to different platforms very difficult (e.g. Windows support). Incidentally, Yaron's ACM Queue article is a great introduction to OCaml.

So as you can see, it has been a busy few months! Much of the core of Mirage is settling down now, and we are writing a paper with detailed performance benchmarks of our various backends. Keep an eye on the Github milestone for the preview release, join our new mailing list, or follow the newly sentient openmirage on twitter!

We've been plugging away on Mirage for the last few months, and things are starting to take shape nicely. As the older blog entries were out-of-date, we have shifted the descriptive material to a new wiki section instead. What else has been happening?

• The Xen microkernel backend is fully event-driven (no interrupts) and very stable under stress testing now. The TCP stack is also complete enough to self-host this website, and you can try it out by navigating to xen.openmirage.org. The stack doesnt actually do retransmissions yet, so your user experience may "vary". Check out the installation and hello world guides to try it out for yourself.

• Richard Mortier has put together a performance testing framework that lets us analyse the performance of Mirage applications on different backends (e.g. UNIX vs Xen), and against other conventional applications (e.g. BIND for DNS serving). Read more in the wiki here.

• Thomas Gazagnaire has rewritten the website to use the COW syntax extensions. He has also started a new job with OCamlPro doing consultancy on OCaml, so congratulations are in order!

• Thomas has also started integrating experimental Node.js support to fill in our buzzword quota for the year (and more seriously, to explore alternative VM backends for Mirage applications).

• The build system (often a bugbear of such OS projects) now fully uses ocamlbuild for all OCaml and C dependencies, and so the whole OS can be rebuilt with different compilers (e.g. LLVM) or flags with a single invocation.

There are some exciting developments coming up later this year too!

• Raphael Proust will be joining the Mirage team in Cambridge over the summer in an internship.

• Anil Madhavapeddy will be giving several tech talks on Mirage: at the OCaml User's Group in Paris this Friday, at Acunu in London on May 31st, and at Citrix Cambridge on June 3rd. If you are interested, please do drop by and say hi.

• Verisign has supported the project with an Internet Infrastructure Grant.

• David Scott (chief architect of the Xen Cloud Platform) and Anil Madhavapeddy will give a joint tutorial on constructing functional operating systems at the Commercial Users of Functional Programming workshop in Tokyo, Japan in September.

Welcome to the new self-hosting website for the Mirage project! As we go about preparing a release for later in the year, this blog will contain technical musings and work-in-progress reports of various bits of the operating system as they mature. Since there's so much to talk about, we decided to start with a blog format, and eventually collect things into a proper document as they stabilise.

Feel free to subscribe to the Atom feed to keep up-to-date with our progress, or just e-mail us or comment on individual posts with any queries.