ArcanOS PRE-MAGUS: frame allocator rudiments

It’s been entirely too long coming, but as of ArcanOS commit e1a75edd4b0a, the rudiments of the frame allocator are available.  Those who’ve followed the project will note that the memory manager was disabled and still hasn’t come back, but this will be coming along very soon.  Really, “memory manager” was a misnomer anyway, as the real goal was to have a basic malloc( )function.  Being able to summon up a pointer to an arbitrary chunk of memory could be considered “handy” for early development.  So, the original allocator is what I’ve been calling a “scratch pad allocator.”  Basically, it just assumes there’s a spare megabyte of RAM located just beyond the kernel’s load location and it uses this area for serving memory requests.  This actually was a pretty neat thing to have on hand in the early days of ArcanOS, before we were using paging and when it was more important to me to have a robust kernel debug log than it was to do anything else.  Of course, this had to be retired when ArcanOS switched over to paging, because the 1MB “scratch pad” shouldn’t be assumed to be mapped in.  No…the malloc( )function should necessarily rely on the paging system.

There are some distinct phases to go through in making a mechanism for allocating out the available RAM:

  1. Identify the physical addresses of all the RAM in system.
  2. Create a tracking system to know what RAM is free and which is used.
  3. Create a mechanism for mapping unused RAM into the address space of the process requesting it.

Step (1) was nicely done for us already by GRUB and the map is passed in as part of the multiboot info struct when GRUB dispatches to the kernel’s entry point.  To know where all the free RAM is in the system, you need only parse the struct.  Of course, to make use of the information in the struct, it’s handy to also tackle step (2), so that you have somewhere to record this information about available memory.  Since I think it’s probably the most minimal data structure possible (and, therefore, the easiest to read, learn, and explain…and thus in line with a goal of ArcanOS), I have opted for the ever-humble bitmap.  ArcanOS has an intentional design decision to target a very generic 32-bit x86 architecture and thus limits its address space to 4GB.  This is broken down into page-size chunks and each one gets a bit in the bitmap.  A 1 in the bitmap denotes free RAM, otherwise it should not be touched.

I set up the bitmap in memmgr.c in multiple passes.  In the first pass, I set the bitmap so that all the page frames are invalidated:

void frame_allocator_init(multiboot_info_t* mbi, uint32_t kernel_base, uint32_t kernel_end) {
	int i;
	int frames_allocated = 0;
	for(i=0; i<(PHYS_MEM_MAP_SIZE); i++) {
		phys_mem_map[i] = 0;

Next, I iterate through the memory map given by GRUB and mark all the full and available 4KB frames as being available:

   multiboot_memory_map_t* mmap;
   for (mmap = (multiboot_memory_map_t *) (mbi->mmap_addr);
		(unsigned long) mmap < (mbi->mmap_addr) + mbi->mmap_length;
		mmap = (multiboot_memory_map_t *) ((unsigned long) mmap
								 + mmap->size + sizeof (mmap->size)))
	 //_kern_print(" size = 0x%x, base_addr = 0x%x%x,"
		//	 " length = 0x%x%x, type = 0x%x\n",
			// (unsigned) mmap->size,
		//	 mmap->addr_high,
	//		 mmap->addr_low,
		//	 mmap->len_high,
	//		 mmap->len_low,
		//	 (unsigned) mmap->type);
	if (mmap->type == MULTIBOOT_MEMORY_AVAILABLE) {
		 //Kindly note here that we will *NOT* be making use of the "high"
		 //fields here.  The memory manager is current capped at 4GB, so
		 //we will deal only with the lower 32-bits of addresses.
		 //The goal here is to flip on the bits for available RAM.  After
		 //that, we will flip OFF the bits for the areas where the kernel
		 //was loaded and where the initial page tables sit.
		 //The frame allocator is very happy to fail to mark partial
		 //frames as "not available."  This is a good, safe starting point
		 //as it means that the only physical memory allocated out will
		 //"really be there."
		 uint32_t base = mmap->addr_low;
		 uint32_t len = mmap->len_low;
	     //First, advance up to the next page frame boundary.
		 if ((base % PAGE_SIZE) != 0) {
			 base = base + (PAGE_SIZE - (base % PAGE_SIZE));
			 len = len - (PAGE_SIZE - (base % PAGE_SIZE));
	     //As long as there is another frame to allocate, allocate it.
		 while (!(len<PAGE_SIZE)) {
			 mark_frame(base, FRAME_STATUS_FREE);
			 base += PAGE_SIZE;
			 len -= PAGE_SIZE;

And then finally I make sure that the space taken up by the kernel and the initial page directory and page tables is marked as unavailable.  This is important to remember to do…the space you’ve used before turning on your memory manager is, by definition, not available to use.  Otherwise, you’ll accidentally write over your page tables and other stuff you’ve so laboriously set up:

   //Now, time to make sure the page directory and any existing page
   //tables are never again re-allocated.
   uint32_t* page_entry = (uint32_t*)INITIAL_PDE;
   for (i=0; i<1024; i++) {
		uint32_t pde_entry = page_entry[i];
		pde_entry &= 0xFFFFF000; //Page table address is in the upper 20 bits
		if (pde_entry != 0) {
			_kern_print("Located existing page at 0x%x and will mark it as used\n", pde_entry);
			mark_frame(pde_entry, FRAME_STATUS_USED);
   //Finally, mark off the area where the kernel is loaded
   kernel_base &= 0xFFFFF000; //Find the nearest page (rounding down)
   while(kernel_base < kernel_end) {
	   mark_frame(kernel_base, FRAME_STATUS_USED);
	   kernel_base += PAGE_SIZE;

And, presto.  It’s a bitmap of memory.  The process of feeding free frames of memory back into the allocator is now a process of finding a 1 in the bitmap, calculating its physical address, and then updating the page directory and tables appropriately.  When I write that code, I’ll break it down on here.  I want to add that there are some obvious drawbacks to the use of a bitmap in this fashion.  The first is that the bitmap is going to be pretty sparse, meaning that space is dedicating to bits that will never actually be used.  The bitmap is also very minimal in what it can do– it’s a map of available RAM, but if a frame is marked as 0, there’s no way to tell why.  I also suspect that the process of returning frames to available RAM is not going to be as simple as it could be, and I further suspect that requesting large data extents (such as 4KB for making a new page table) may not go in a perfectly optimal fashion.  All of these, however, are hits I’m willing to take in exchange for the fact that it’s a simple, easy to code, and easy to explain data structure.  This shows how to take the information in a GRUB multiboot info struct and use it to build a map of the available page frames in memory, and being able to show this in a single blog post isn’t so bad.

So, the next step is to get the allocator allocating, which is to say the next step is to take frames out of the bitmap and map their physical addresses to available addresses in the kernel’s address space.

Foot Injury

I haven’t said too much about my athletics on here yet, and this seems like as good a place as any to start, even if it is on a somewhat sad note.

For the most part, I’ve had two major activities driving my athletics– cycling and running.  Cycling is my “bread and butter” event.  I’m better at it and I like it more.  It’s also been a very practical activity, in that I can bike to get where I’m going and I get a workout, too.  In fact, most of my training has come in the form of commuting.  But, as my cardiovascular endurance grew, I took up running as a secondary activity.  I wouldn’t say that I’m a great runner, but I’ve done pretty good with it for what’s ultimately been a “back-burner activity.”  Going out and getting a good 6 miles on foot, seeing an area from that point of view…well, it’s just fun.  It’s great to run up the hills of Richmond and look out over the San Francisco Bay.  It’s great to run the back trails around hilltop castles in Cornwall.  And during times when cycling has come to feel like a daily chore, running became a way to keep my joy of exercise alive.

And, I should note, almost all the running I’ve done has been barefoot running.  I’ve been using some form of Vibram Five Fingers for years now.  It’s made all the difference to me, because I used to get pretty frustrating back pain from running, but barefoot running, and the associated changes in gait, took care of this problem.  Barefoot, I’ve run all over Florida, California, and Cornwall.  I was barefoot in my first 5k, and then my first 10k, and then my first Bay to Breakers, and finally in my first half marathon.  I seriously doubt I’d have ever made it this far if I hadn’t made the switch to barefoot.

I mention this not just to shill for barefoot running but because I want to make it clear that I’m no rookie.  I’ve been running barefoot for years and I’ve even turned in some respectable times in my races.  But, now, it seems that my running may need to stop for an extended period of time, and I suspect that I’ve actually been a victim of my own success.  See, when I moved up to the East Bay, I saw the opportunity to use our train system for my commute, but I would need to still travel about 7.5 miles every day in order to cover the distances to and from the train stations.  Due to various reasons, a bike was out of the question, so I decided I’d try strapping on a backpack and running.  This seemed a pretty straightforward thing to do.  The contents of my backpack are only a couple of pounds (a tablet computer, wallet, and clothes, essentially), and running 7 miles is well within my reach, especially when each 7.5 mile day is actually 4 different runs.  And I’d be doing this 4 days a week, since I needed my car on Tuesdays.  Seemed like a great way to become a better runner, and I started to really look forward to the improvements in my times.

And this actually did work out pretty well for a couple of months.  I am beginning to suspect, however, that there were some odd aches and pains that I let slide which I shouldn’t have.  About six weeks ago, I finished my run to work experiencing some difficult pain in the bottom of my heel.  It was an intense but dull ache.  I ran home on it anyway, and then made it worse, and since then, I’ve had to stop running and heal.  After two weeks of being pain-free, I decided to try the daily runs again.  It only took one day of running to produce injury yet again.  Today, I’m limping around the office as the bottom of my foot aches with a strained/sprained sensation that runs all the way up to the base of my calf muscle.  I’m not 100% certain what I’ve done, but plantar fasciitis is my best guess.  I don’t believe that it’s been caused by a single acute episode but instead is the result of ongoing stress on my plantar fascia.  A likely story is that I may run with my heels a bit high and that this, coupled with a little extra weight from the backpack, has excessively stretched my plantar fascia.  No individual run caused the damage, but instead it’s been death by a thousand cuts as it’s crept up on me over time.

This is very frustrating, though, since it looks like I will have to back out of the Big Gay 10k, which is one of my favorite runs.  I also am going to have to seriously consider that I won’t be able to run in the San Jose Rock and Roll Half Marathon, either.  Normally, I’d make up the slack in exercise with cycling, but my trusty road bike, Thumper, was stolen a few weeks ago and the only cardio equipment in my home gym is an elliptical machine, which still involves a bit of foot flexing.  High intensity interval squats may be in my future for a few weeks.  This is probably a good excuse to surf more and to buy a new bike.

What’s not clear to me, though, is what to do to heal sooner and prevent this in the future.  I should see a doctor, but I’m too busy (and you’ll find out why soon enough), so I’m just trying to take it easy.  It’s likely this was a chronic problem so healing may take a very long time.  I’m also not sure if I’ll be able to return to barefoot running without running another injury risk.  If I can’t, then it quite likely could be the end of my running “career,” because I find no pleasure in running in shoes.  I ran barefoot as a child, and I take too much direct pleasure in feeling the ground beneath my feet.  I should consider, however, that it is constant barefoot running that led to this.  I have no arches in my feet, and that much engagement with my forefoot might have stretched some ligaments more than they can endure.

The Long, Hard Road out of Segmentation

As of last night’s Pre-MAGUS check-in, ArcanOS now uses paging rather than segmentation.  Of course, this is a pretty critical step in the MAGUS goals.  Segmentation is really just a legacy aspect of x86 these days, and there are really no meaningful examples of its use in modern OS development.  So, why did I ever use segmentation in the first place?  Well, the biggest reason is that I inherited it.  Remember that ArcanOS began life as an early coursework lab for a teaching operating system called JOS, and very specifically an early introductory form of it that focused on the boot loader.  ArcanOS, like JOS, and like a lot of other operating systems, is designed to be a “higher half” kernel, which means that it is linked to 0xC00000, but it’s actually loaded at a much lower address (right at 1MB, for those playing along at home).  If you really want to split hairs, ArcanOS is linked at 0xC0100000 (which will change soon, possibly in the next check-in).  In order to really “run” ArcanOS, the x86 processor has to be set so that address 0xC0100000 is mapped to 0x00100000.

This was originally done with something the hobby OS community generally calls “the GDT trick“.  In order to run in 32-bit mode, an x86 requires you set up a GDT, and this will set the values of the segment registers.  These registers will be added to every address to get a final result that goes out onto the memory bus.  It turns out that, in this addition process, overflowing 32 bits will just cause a wrap-around.  So, you can exploit this to get your kernel’s link addresses to refer to its actual load addresses (0xC0000000 + 0x4000000 = 0x00000000 after wrap-around).  It’s good for getting things up and running, but you can’t really stake much on it after that.  Segmentation is inflexible and you can’t do any swapping with it.  On top of that, there are all sorts of memory addresses that correspond to physical hardware, and you have to remember to adjust those addresses to compensate for your segmentation trick.  Even more so, though, it seems that the GDT trick isn’t universally supported.  ArcanOS uses Bochs as its reference platform, but it would appear that VirtualBox doesn’t support the GDT trick.  Long story short– getting onto paging is pretty important.

So, this left me with some questions about how I wanted to get onto paging.  My first thought was to employ the GDT trick to jump into my proper kernel code, then set up paging and reload the GDT with segment descriptors that used a base of 0x00000000 (effectively turning off segmentation).  The more I considered this, though, the less I liked it.  Part of the reason ArcanOS started out using the GDT trick was because it also started out with its own boot loader and you have to set up a GDT to get into 32-bit mode in the first place.  ArcanOS now uses GRUB for its boot loader, though, and GRUB already sets up 32-bit mode and installs a GDT to support a flat memory model.  Why was I going to change the GDT only end up setting it back the way it was before?  No, this was a good opportunity to separate ArcanOS from its roots by just going straight to paging.

Setting up paging, it turns out, is a little bit easier than I thought.  You need about 12KB of memory to establish some initial page tables, and if you’re loading the kernel at 1MB, there’s usually plenty of room just under the kernel to use for those purposes.  After that, it’s really just zeroing-out the memory and stuffing some values in the tables.  Info on the layout of the tables, with reference for making a higher half kernel, is in an article on the OSDev Wiki.  The sticky wicket, in my mind, was that I would really just rather not write it in assembly.  I don’t mind writing in assembly, but for anything non-trivial (especially in a hobby project where my free time is at a premium), I trust a compiler more.  But…if I wanted to do this in C, wouldn’t I have to write it in the kernel proper, thus requiring I rely on the GDT trick?  As it turns out, the answer is “no.”  What I realized is that the code would be location-independent as long as all I did was write some simple loops and not refer to any memory except the page tables which I was trying to fill.  All of the control flow would be PC-relative (based on adding offsets to the current code location), so it would never care where it was really located as long as I called into it correctly.  The result is the init_paging( ) function in kernel/memmgr.c.  All I have to do to call the function is to store its address in a register, adjust its link address to its load address, and then call the function from the register.  I call this function, let it do its thing, and then I finish up setting paging in assembly.

Originally, I paged the first megabyte to itself and paged 0x00100000 to 0xC0100000.  This makes the hardware exposed in low memory easy to access, ensures my first page tables are mapped in, etc, and it ensures the symbols in the kernel were all mapped in.  And, then I set up paging and….crash.  Why?  Well, because I’m already running my code at 0x00100000…which is in the second megabyte.  The address of the current instruction was not mapped to anything!  So, once paging was turned on, my own code became inaccessible.  In the end, I ended up deciding to map the first four megabytes to themselves.  I’ll want to clean that out eventually, but for initialization, it’s something I can live with.  So, this was enough to get me into the ArcanOS kernel proper, but then…crash.  What was it this time?  Well…GRUB put the multiboot info struct below 0xC0100000, so that wasn’t mapped in.  Again, just to keep the bases covered, I mapped the first four megabytes to the address space starting at 0xC0000000.  This does mean there are currently two ways to access the same 4MB of memory, but it gets ArcanOS up and booting, and there should be some easy ways to clean this up.

Technically, with this view of memory, ArcanOS would have run just like the GDT trick was still being used, but I still did due diligence and I went through and stopped referring to addresses as being relative to KERNBASE.  My word of advice to any OS hobbyist who’s starting out– I really encourage you to consider doing rudimentary paging right from the start.  If you don’t, you’re going to get married to an address management scheme you’re likely to throw away later, and if you’ve been using it for a while, then it’ll take a harder portage effort to get off of it.  Just page your memory in from the get go.  If you’re using GRUB, even more to the point, because GRUB already set up the GDT for you to use paging with a flat memory model.  The biggest concern I had with setting up paging was that it looked hard at a time when I just wanted to get a kernel booting and doing some interesting things.  In reality, I was able to sling together a C function which would set up the page tables, and this made things very easy indeed.  Don’t get into using segmentation.  If you do, it’ll be a long, hard road getting back out.

So, from here, what I really need to do is clean up the paging a little bit.  Once I’m jumped into the kernel, I can stop identity-paging the first 4MB.  I also want to map only enough pages to hold the kernel.  Also, I don’t see a good reason to link the kernel at 0xC01000000 any more, since doing that was mostly to make the GDT trick a little bit simpler.  I have a good map of physical memory thanks to GRUB and I can make and manipulate pages, so this is everything the memory manager needs to be pretty full-featured.

Perry v Schwarzenegger (Prop 8 Trial) Motion to Vacate Denied

I have been waiting all day to type that headline.  I deleted it and then typed it again.  It felt good both times.

Here’s a link to the order.  It contains a lot references you can follow if you want to go into the case law behind what’s written here.

In reality, we’re talking about a fairly small tributary of the long river that is the Perry v Schwarzenegger case, which is fairly commonly called the “Prop 8 case.”  The case is in a bit of a holding pattern as it waits in appeal in the 9th Circuit.  The 9th Circuit has certified a question to the California Supreme Court to determine if the intervening defendants in the case, broadly called “the Prop 8 proponents,” actually have standing.  I’m actually not going to go too much into that aspect of things today, though it would come as a surprise to nobody that I have opinions on the subject.  I just mention the state of affairs to set the stage a little.  Look for something interesting on that front in (hopefully) early autumn.

No.  Today we’re talking about an extra motion that was filed while everyone waits to wrestle with the question of standing.  The judge which presided over the case, Vaughn Walker, made a public statement this past April stating that he has been in a same-sex relationship for the past decade.  Proponents of Prop 8 subsequently filed a motion to vacate Walker’s judgment.  As the name implies, it’s a request to throw out the judgment.  The basis for this motion was that Walker, being in a same-sex relationship, had an obligation to disclose his relationship and, ultimately, recuse himself from the case.  I’m fairly sure that most people who’ve followed the case considered this motion to be fairly frivolous, but I still want to dig into it a little bit because I think it helps to clarify some things about judicial impartiality and the integrity of the judiciary.

It’s important to understand that the starting point, and indeed the default state, is the impartial nature of a judge.  Impartiality is part of the oath of a judge, and it’s affirmed repeatedly through case law.  In fact, impartiality is an essential component of maintaining a nation of laws in the first place, and if no officer can be found who stands by the law first and foremost, then the whole system falls apart.  Therefore, it’s important that judges be treated as impartial and that rules be established that allow the transparent testing for partiality in the minority of cases where a judge genuinely cannot be trusted to act in an impartial fashion.  To that end, we have something commonly known as Section 455 which lays out criteria on which a judge may be disqualified.  This motion to vacate focuses on two aspects of Section 455, which I’ll include here:

(a) Any justice, judge, or magistrate judge of the United States shall disqualify himself in any proceeding in which his impartiality might reasonably be questioned.

[…](b) He shall also disqualify himself in the following circumstances:

[…](4) He knows that he, individually or as a fiduciary, or his spouse or minor child residing in his household, has a financial interest in the subject matter in controversy or in a party to the proceeding, or any other interest that could be substantially affected by the outcome of the proceeding[.]

The proponents of Prop 8 challenged Walker’s impartiality based on the fact that he announced he was in a long-term same-sex relationship, citing violations of Section 455(a) and Section 455 (b)(4).  Let’s look at each of these in turn.

Section 455(b)(4) disqualifies a judge based on his or her potential to have a financial interest in a case, and that form of conflict of interest is undoubtedly the sort of thing that you try to avoid in an orderly and fair judicial system.  For example, if a judge (or the judge’s family) owns stock in a company involved in a case, then there’s a clear conflict of interest.  Other interesting cases that have come up in the past include things like the judge’s family possibly being in the class of a class-action suit.  Of the options available for disqualifying a judge, this is one of the more objective ones that can be put forward.

In the live blog of the oral arguments, you’ll see the attempt at pinning Walker’s relationship status to Section 455(b)(4).  Essentially, it goes like this: Walker’s in a ten-year relationship, which means he wants to get marries, which means that he wants to enjoy the financial benefits of marriage, which means he has a conflict of interests.  To put it another way, Walker isn’t capable of being impartial in a trial about gay marriage because marriage is financially advantageous and he could, if Prop 8 were ruled against, marry his partner.  Of course, this breaks down on a number of fronts.  There’s no evidence of intent to marry, for example, and there’s no reliable way to test it.  There’s the fact that marriage requires two people who want to marry and not just one, so you’d also have to test Walker’s partner.  There’s the fact that people change their minds quite a bit.  The fact that the Prop 8 proponents took the intent to marry as given ultimately broke their chain of logic.  See pages 9-10 for what I consider the more bash-it-with-a-hammer part of the order.  I’d quote it, but really…just read the whole thing.

So, this leaves Section 455(a).  The language for this rule requires a little bit of clarification.  It does not leave the door open to personal speculation, even if it seems “reasonable” for the party doing the speculation.  Instead, there is a test known as the “reasonable person” test that helps to make concrete the concept of Section 455(a) and to provide a test.  I’ll quote from the order:

In this context,the “reasonable person” is not someone who is “hypersensitive or unduly suspicious,” but rather a“well-informed, thoughtful observer” who “understand[s] all the relevant facts” and “has examined the record and law.”  United States v. Holland, 519 F.3d 909, 914 (9th Cir. 2008) (citations omitted).15  This standard does not mandate recusal upon the mere “unsubstantiated suspicion of personal bias or prejudice.”  Id. (citation omitted). […] In addition, the Court recognizes that a fact is not necessarily a basis for questioning a judge’s impartiality merely because that fact might lead a segment of the public to question the judge’s impartiality.  Reasonableness is not determined on the basis of what a particular group of individuals may think, nor even on the basis of what a majority of individuals in a group believe to be the case.

This is something you see in law from time to time.  In order to provide a test of a concept, a sort of hypothetical person, with certain qualities, is conjured up and a discussion is held about what this person would conclude or know.  These constructed people are a method of trying to project the concepts of law into a hypothetical third party whose thoughts the parties in the case may discuss.

Very similar reasons were given for why Walker should have been dismissed under Section 455(a): he’s in a long-term same-sex relationship, which means his impartiality is in question.  To really reach this point, it would have to be demonstrated that his personal desire to marry his partner would have to be sufficient that he could not preside over the case in a professional and impartial fashion.  Again, however, there is no fact that can be pointed to in this instance, since the Prop 8 proponents are again basing this conclusion on the fact that he is in a long-term partnership.  If you review the live blog (linked above) at the 10:39 timestamp (and the ones slightly earlier), there is a distinct pattern where it is concluded that having a long-term partnership immediately implies a powerful desire to marry.  Walker never said one way or the other, though, and his silence, absent of any other evidence, can’t be construed as proof of partiality.  In the court’s order denying the motion to vacate, it exposes the crux of the issue:

A well-informed,thoughtful observer would recognize that the mere fact that a judge is in a relationship with anotherperson–whether of the same or the opposite sex–does not ipso facto imply that the judge must be sointerested in marrying that person that he would be unable to exhibit the impartiality which, it is presumed, all federal judges maintain.

Things might have been different if Walker had issued remarks in the public record about an obsession with marriage, with how the case was the only thing standing between him and being married, etc.  It’d show a massive emotional investiture in a major outcome.  He didn’t.  Furthermore, there is a lingering suspicion on the part of the proponents of Prop 8 that all same-sex couples must be practically obsessed with their ability to marry, to the point that Prop 8 is the only thing stopping them.  I’m pleased to see sexual orientation invoked in the “reasonable observer” test, because one can point to many unmarried long-term opposite-sex couples.  Why would a same-sex couple be any different?  People choose to marry for their own reasons in their own time.

That really seals it, but there are some extra parts with respect to the argument that Walker’s failure to disclose creates reasonable doubts about his impartiality.  Such an argument is, effectively, flipping the script.  It makes the assumption that, should you find out something about a judge you consider a threat to impartiality, the judge must have already known this, too, and decided to remain silent on the subject.  Silence, however, does not imply motive.  You can easily make an alternative motive the the one presented– the judge considered everything and concluded that s/he didn’t have any threats to being impartial.  Silence cannot be considered automatic evasion, and to accept this argument basically means that it’s possible to accuse a judge of being impartial whenever one of the parties finds something that s/he doesn’t like.  This completely subverts the system of presuming judges can take their oaths seriously and creating rules for challenging that assumption.  To adopt such a stance would be damaging to the integrity of the judiciary:

Contrary to the intent of Section 455, which was designed to preserve judicial integrity through practices of transparency, it is clear that fostering the practice of commencing a judicial proceeding with an extensive exploration into the history and psyche of the presiding judge would produce the spurious appearance that irrelevant personal information could impact the judge’s decision-making, which would be harmful to the integrity of the courts.

And that pretty much sews it up.  I see this not only as a positive result for marriage equality but a major re-affirmation of judicial integrity and impartiality in light of questions about sexual orientation, which itself is somewhat novel and relevant.  In addition, and this could just be my own armchair quarterbacking here, I think that the motion to vacate was an intentional tactic on the part of the Prop 8 proponents.  It’s been many months since I read Walker’s judgment, but I seem to recall it having a large number of findings of fact that leave the proponents’ case significantly wanting.  Under a de novo standard of review, the 9th Circuit would be able to provide an independent conclusion regarding questions of the law in the ruling but they would be working with the lower court’s records and findings of fact.  While I found Walker’s application of law in the case fascinating and damning, I could envision debate about it.  By comparison, the findings of fact will be the basis for review, and the Prop 8 proponents don’t want that.  Their only hope is to vacate the judgment and get a new trial and better facts.  That tack failed.

I do believe this is good all around.  Aside from any further motions, the next battleground will be regarding standing.  I’ll, hopefully, be back in autumn on that.

ArcanOS PRE-MAGUS Update

With most of the “fun” of buying a house and moving to a new part of the Bay behind me, I’ve finally had some leisure time to turn my attention to something I’ve really missed doing…namely, developing ArcanOS.  What have I been up to on that front?

Well, after having been off the project for several months due to the whole “becoming a homeowner” thing, I took a not-so-long look at the project and concluded that I needed to be more realistic about the boot loader situation.  When we last left our hero, he was intending to continue development on his boot loader until such time as it could produce a working memory map, at which point ArcanOS would be ported to the Multiboot Specification so that it could be booted by GRUB.  While I still think it would have been a lot of fun to do that, I ultimately had to be realistic about what I was getting myself into.  I’d be building multiple boot loader stages, making a new build process to support them, and enjoying the hair-pulling process of chain loading and making sure I hold onto the memory map in the process.  That would have been a good learning experience, but it would take a lot of time to do and I already lost a lot of momentum over the past few months.  Doing a lot of work only to throw it away and do a portage effort just didn’t look as appetizing.

So, as of a recent commit, ArcanOS now boots via GRUB.  The current commit will generate a disk image file for Bochs which contains the GRUB stage1 and stage2 boot loaders already.  As of tonight’s commit, ArcanOS now receives an accurate memory map from GRUB and this will give me the material I need to make a frame allocator and then work on enabling paging.

Part of me is tempted to take a side trip and include some bells and whistles to make the boot process easier.  I’m getting tired of typing in the block list of the kernel on every test boot.  I’m fairly sure that much of that sort of thing, though, requires a file system GRUB understands, and if I leap onto a canonical file system, I might never get out.  I should also note that my build process needs some tweaking…right now I have to regenerate the file in boot/post_pad every time the kernel size changes!

Since I’m back to proper feature development for the MAGUS goals, I’ve changed the version name to PRE-MAGUS.

AIDS/LifeCycle: Advice From a Veteran

It’s the time of year when mad cyclists turn their eyes to San Francisco, where they will begin the week-long challenge known as AIDS/LifeCycle.  For personal reasons, I opted to take this year off from the ride, though I am giving very serious consideration to returning to it next year.  My friend, Erin “Blast” Coplan, will be stepping into the saddle soon, though, and while I know she’ll be a total champ out on the ride, I wanted to pass along some things that I learned from my ALC ride last year…stuff I wish I’d known on day one because it would have meant a more comfortable trip.  With that in mind, Blast, here’s some little things to keep in mind as you make your final preparations, and please remember some of this for the road!

  • If you read nothing else, please read this– Have a recovery plan and execute it the moment you rack your bike. This was, ultimately, my biggest mistake.  I would get off my bike, drink some hydration formula, and then promise myself to have a snack after my tent was pitched.  That’s not a recovery plan.  You’ve got an hour, once you’re off your bike, to pump protein back into your body and start repairing your fatigued muscles.  You must do this.  Failure to do this will make progressive fatigue set in, and come the latter half of the ride, you’ll be digging yourself out of a hole.  Keep a packet of recovery drink mix on you.  Don’t wait for gear check.  Don’t wait to pitch your tent.  Don’t wait for a shower first.  Start recovery immediately after racking your bike.
  • Chamois cream is your friend.  Use it religiously.  Put some on when you start each day and add a little more at each rest area.  Yes, it’s goopy and slick and it can make your chamois feel goopy and slick, but it beats the alternative, which is getting a sore.  This happened to me.  I thought I didn’t need chamois cream because, in all the century rides I’d done, I’d never had a problem.  Even well-fitting shorts and a good chamois aren’t always enough.  All it took was one seam pinching some damp skin and I earned a sore I had to treat all week.  If I’d simply used chamois cream from the beginning, I wouldn’t have had this problem to manage.
  • Riders put a lot of effort into being ready to go up hills.  Please be really mindful of how you go down hills.  You don’t hear very much about people having serious injury crashes on an uphill.  ALC has some fairly steep descents in it, and while they’re fun, don’t be afraid to feather your brakes and stay within a safe operating speed for you.  It takes only one bad crack in the pavement or one swerve to avoid something and now you’re falling down a hill at 30 mph.  Going fast is fun, but if you’re going faster than you know how to handle, you’re taking a risk.
  • You’ll start Day 1 riding through San Francisco.  San Francisco is often very foggy in the morning, and you’ll also be doing a bit of climbing in the city, putting you up in what can be potentially thick fog.  At this point in the day, the pack of riders will not have had a chance to thin out.  Keep a little extra distance between you and other riders, because hazards can come popping out of the fog.  If it’s foggy, watch out for water beading up on your eyeglasses, because it will mess with your vision.  Take it easy…day 1 is a pretty long one.  Save yourself for the climbs on the way to Santa Cruz.
  • The Clif Shot Blocks in the blue pouches have caffeine in them.  This is important to know because the coffee at camp is famously weak.  Caffeine is an utterly magic thing to have on long rides, especially early in the day.  Not only does it give you alertness, but it’s mood-elevating and helps hide those casual aches and pains.  Get yourself a good little hoard of those caffeinated Shot Blocks, because they’re powerful medicine and will disappear later in the week when other people start hoarding them.  Also, unless you want to blow a night’s sleep, don’t eat them in the afternoon.
  • Figure out a good daily rhythm with your tent-mate.  I, personally, wanted to hit the trail very early every morning, so my tent-mate and I had a rule.  I would pitch the tent and he’d strike the tent.  This was a great arrangement for both of us because it meant that, at the point in the day when we’d need it the most, the other one would be taking care of the tent.
  • Take it easy on yourself for an hour after lunch.  You don’t have to sit around at the lunch break for a full hour, but do remember that digesting solid food redirects your bloodflow to your stomach and you’ll have less energy for pedaling.
  • For the love of all that is holy, sunscreen your lips!  Put fresh sunblock on at every rest area, even if it’s not all that sunny.  You do not want to be one of those people at the end of the week…your swollen and zinc oxide covered lips making you look like some incredibly sad, spandex-clad clown.
  • The easiest way I know to insure you ride just below your comfort zone– keep riding until you find someone that you’re just barely overtaking.  Ride behind that person.  You’ll make a new friend and you’ll ride just a bit slower than you naturally would, so you’ll have energy left over for climbs and wind gusts.
  • Day Two is not only a long ride.  It’s also a hot ride and it can be windy enough to require you to set a slower pace.  You’ll be between rest areas for longer and doing a fair amount of work.  Make sure you keep your water bottles topped up and make sure to use them heavily, even if that means stopping.  If you go to the medical tent suffering from heat exposure, you may end up losing a day on the ride while they rehydrate you.  Keep liquids in your body and pay close attention to this on days where you’re inland.
  • Two things about hills.  Thing one… Quadbuster and the Evil Twins are the “legendary hills,” but pretty much every day will have a climb that will make you question your choice in hobbies.  Personally, I thought the worst climbs of the ride were actually on Day Five.  They’re not long, but they become suddenly steep and you’ll already be shagged out from the Evil Twins the day before.  Thing two… When you look up at these hills, and you see the road turning up ahead, don’t get all excited and tell yourself that the top of the hill must be up ahead.  You’re on the hill until you’re through with it.  Dial in your climbing gear (I’m not too proud to say that I climb on a “granny gear”), get in a pace you could hold indefinitely, and go with it.  You’ll still finish the day, and you still get to feel like a badass when you finish the hill.
  • While I’m mentioning Day Five, it was the most challenging day for me.  The trail map makes it look like everything’s pretty easy-peasy after lunch, but that’s when the winds started.  Relax, don’t let yourself get down, take it easy, use your rest areas, and just focus on putting miles behind you.  You’ll get to camp.  Just don’t get caught up in the wind and the terrain.
  • I never understood the value of a good massage until I was on ALC.  One of my knees had its ACL replaced some years ago, and I discovered that many days of riding can make that knee feel a little “tweaky,” most likely because the the small muscles around the knee get sore.  Getting a massage on that leg was like erasing a day’s fatigue.  Camp will give you a free massage; use it.  You can probably buy one when you pass through Bradley, so bring money.  You can probably get another one in Santa Barbara.  They’re all worth taking time off the ride to get.  Something will be in danger of becoming chronically sore, and the trick to ALC is prevention.
  • There are a lot of opportunities to socialize at night, but remember that your first job is getting a full night’s sleep.  Day over day, you can’t ride tired.  I’m probably a fairly extreme case, but I was in bed at roughly 7:00 PM every night.  Waking up before dawn, having had a full night’s sleep, is an amazingly refreshing thing.
  • Start the day with a dose of acetaminophen or ibuprofen, and then keep using it through the day.  Don’t wait until you’re sore.  They’re anti-inflammatory drugs.  Stay ahead of the inflammation and you stay ahead of both pain and damage.
  • Keep supplies on you to replace both of your tubes at any point.  Otherwise, you may end up getting sagged.
  • If you do get sagged, remember that it happens to everyone.  We all pull something, break something, or overdo it.  Discretion really is the better part of valor, and it’s always better to ride another day than to not.
  • Remember that, no matter what happens on the ride, you’re out there because you’re a hero.  This ride is your celebration, not your burden, so go out there and enjoy it.

I guess that’s about it.  If this is Blast reading this, then good luck and may the wind be at your back.  If you’re not Blast, why not consider contributing to her ride?

In Review: Apples/Blossoms

In Review: Apples and Blossoms by Juliana Finch

For fans of Juliana Finch, one of America’s great “undiscovered” female singer-songwriters, 2011 couldn’t possibly come soon enough.  Since her 2007 release “How to Take the Fall,” a powerful sophomore production and fitting follow-up to her seminal “Omelettes and Biscuits,” she has hinted at release of new material through a number of live shows, online pre-releases, and a strong bid at the (sadly, canceled) Lilith Fair Atlanta.  Inarguably, “How to Take the Fall” was a reminder that the traditional female singer-songwriter formula survived the onslaught of the late 1990s intact, and Finch’s unique blend of folk, country, blues, bluegrass, and Irish traditional music has been a stand-out example that the American folk canon is still out there, waiting for another bright mind to pick up some of its fragments and melt them together into a unique and beautiful sound.  The question has always been one of where Finch would go next.

After four long years, during which “How to Take the Fall” has remained timeless and classic, Finch has followed up with “Apples” and “Blossoms,” a diptych of EPs intended to follow the seasons of the year.  The cover art alone, when compared to “How to Take the Fall,” tells a story of just how much Finch has grown, both personally and musically, in that period of time.  Far from the realist imagery and symbolic heartbreak of “How to Take the Fall”, “Apples/Blossoms” features simple, colorful depictions of a tree in autumn and spring.  The image is personal, from the “finch bird” nesting in the tree to the “JF+GH” initials, indicative of Finch and her spouse.  The great success of “Apples/Blossoms” is in its very intentional decision to not be the massive magnum opus of its predecessor.  The EPs represent a very intentional choice of a seasoned musician who, having broken down the door already, now taking time to enjoy the room.  It’s this level of personal expression and exploration that makes “Apples/Blossoms” so eminently wonderful to experience.

“Apples” opens the duo, and in many ways, seems to pick up where “How to Take the Fall” left off.  The lead track, “Say the Word,” is very much arranged and mixed in the style of a track from Finch’s previous works, and carries with it that distinctive sound that immediately reminds the listener of a glass of old bourbon (neat, of course), flaking paint on a weathered front porch, and the cold zephyrs of Stone Mountain around sunset time.  It carries familiar themes of sentimentality, love, and loss, always wrapped in Finch’s trademark vocal style…fragile, yet never maudlin, and kept warm, rich, and world-wise with appropriate accents of blues and jazz elements.  One of Finch’s great achievements in songs like “Say the Word” is her ability to draw the listener into the emotional content of the song but to leave no room to wallow.

To think that “Apples” is completely a bridge from prior works, however, is to vastly miss the point.  Perhaps it’s just that autumn is that time when we all contemplate the one that got away.  It is also, however, a time to contemplate matters of the home, of things ending and beginning, and to see the treasure in those we love the most.  This is where Finch makes her greatest artistic steps in “Apples.”  Where “How to Take the Fall” was a study in heartbreak punctuated by moments of relief and vague optimism, “Apples” revels profoundly genuine promise and the joys of the moment.  It is, ultimately, a picture of a happy, warm home filled with love.  Whether it’s the feeling of waking up tangled in the sheets with your beloved, looking around at the walls of your home in satisfaction, or the hope and promise that comes with Christmas, the solstice, and the lengthening of days, it’s impossible to not hear in Finch’s voice someone whose heart is truly and deeply at rest.  For those of us who know that feeling, “Apples” is a powerful reminder; for those who might not yet, it’s a powerful blueprint.

“Apples” finishes on a wonderfully humorous note, lightening the mood and self-reflection with a cover of “Walking After Midnight” recorded live.  Finch’s jazzy chops and multifaceted vocal prowess are, of course, beyond sufficient to the task of cutting the EP’s sentimentality before it drags the listener too deeply into reverie, and the track provides an excellent breaking point for “Blossoms” to step in.

“Blossoms” opens with a cover of “All of Me,” arranged with guitar and accordion in a fashion that would likely make Jacques Brel smile if he were capable.  It’s nearly impossible to reiterate how clearly comfortable Finch seems with jazz, and one can only hope this may be a new vista ripe for her exploration, as various touches of jazz grace “Blossoms” through and through.  Diehard fans of Finch will be delighted to have a studio recording of “Peachtree,” one of the most promising songs from her live performances.  What will be a complete surprise, though, is the song’s arrangement, using upright bass, electric piano, and a dash of percussion to give it a bit of a throwback feel.  Where the live performance of “Peachtree” is the very typical bright-and-tender sound we’ve all come to expect, the studio version is warm and a little jazzy, something you’d expect from the Greenwich Village less than from the actual Peachtree St in Atlanta.

Of course, “Blossoms” is all about spring.  The sunlight shines through in every track.  The cherries and apples and dogwood are blooming while the leaves are still in their buds.  Finch, again, turns to love and human connection for her inspiration, and again, it is full of promise and hope where “How to Take the Fall” was full of reflection and longing.  Where “Apples” seems directly focused at one lover…perhaps a life partner…and about home, “Blossoms” hops and skips around, exploring first the feeling of infatuation, then of professing to be in love, and finally of commitment and marriage.  Where “Small Things” leaves off with a June bride, “What You’re Wishing For” on “Apples” picks up with, perhaps, a first family Christmas.  “Blossoms” caps off with the upbeat and light-hearted “Radiator Charlie,” which again helps to clear off the intense emotionality of the EP before the listener can get lost in thoughts and memories.  Those who adore Finch’s mastery of country and bluegrass will be in for a treat with this track.  Besides…it’s spring, and it’s time to stop dancing in the kitchen and start dancing in the yard!

Finch’s third major release is in every way what we’ve come to expect from her.  There’s no pretense in her musical career…she takes timeless and evocative raw material and, through careful arranging and her signature sweet-as-molasses voice, refines something far beyond the whole of its parts.  What’s so ultimately rewarding about “Apples/Blossoms,” however, is its power to come across as deeply intimate.  This isn’t Juliana Finch The Musician.  It’s Juliana Finch.  Or maybe just “JF,” like it says on the tree.  Through both “Apples” and “Blossoms,” there’s an unavoidable impression that Finch has seen so much more of the world and of herself since “How to Take the Fall.”  There’s a certain settling-into-herself with the release of “Apples/Blossoms,” with songs that may not have explicitly been written for these albums (“Small Things” was a gift at a friend’s wedding), but which seem to give off a story of someone leaving behind the slings and arrows of dating life and emerging into something more.  If we’ve all had to wait four years while she’s made that journey, then it’s been four years well-spent for all of us.  Let’s just hope we don’t have to wait another four before she blows us away again.

Introducing the HP TouchPad

I’d like to take a moment to introduce to you the project that has been taking up a considerable amount of my time and attention lately.  Here it is:

HP TouchPad

Hi. I'm the TouchPad.

This, my dear friends and readers, is the HP TouchPad, a cornerstone of HP’s new line of WebOS products.  Since HP’s acquisition of Palm last summer, I have been working on the internals of WebOS, the operating system which has powered Palm’s current line of devices, and I’ve also been hard at work bringing new features to help make the transition from smart phones to a tablet a pleasant one for both developers and users.  Officially, I’ve been working on the windowing system (known as “cards” and “card stacks”), but the reach of my efforts has extended into the native applications and gaming.  I’ve also been a journeyman bug fixer during my transition into the Palm organization, before finally settling into feature sets I can call my own.

I’m very, very excited about the release of the TouchPad.  I have to admit that I don’t really think people have seen the potential of WebOS until now.  Phones are just not the ideal place to demonstrate the features of a fully multitasking operating system.  The limited amount of screen space and limited amount of attention span devoted to one’s phone just don’t add up to the kind of rich, long-lived interactions which benefit the WebOS model of concurrently running applications.  I do, however, think that this model can seriously up the game in the tablets market, which is currently dominated only by iOS and which will only become a truly competitive market this year as the TouchPad enters the fray with both iPad and Android products.

The TouchPad was announced today, along with the HP Pre3 and HP Veer (both of which I have also been an active contributor to), at a major demo in San Francisco.  I really think the demo went quite well, with only a small hiccup.  It looks like the hands-on demos afterward likewise went well.  I haven’t been able to find video from the formal on-stage demos yet, but here are some great videos of our official features and a hands-on:

Of course, this tablet is “my kid,” so I’m going to have opinions about how great I think it is.  Either way, though, this is a very big day for me personally.  You see, I’ve always had a dream, ever since the age of five when I first put my hands on a keyboard, that I would one day “make the computers go.”  I always dreamed not of being an applications developer but of a designer of the machines which people would use.  I used to lay awake on summer nights in my home in central Florida, laying on top of the blankets, sweating lightly in the thick air, and think about how, one day, I’d know enough about what was going on in that multitude of silicon to the point that I would make it come to life.  That dream is what sent me off to college to learn computer science, and it’s what sent me to graduate school when my undergrad years failed to teach me enough to fulfill my dream.  Since leaving school, I’ve been fortunate enough to work in the mobile industry where I’ve had the chance to work on phones and netbooks, but every project of mine has had a feeling of ill fate to it.  The products never gained the level of attention that made me feel like I’d put something out there in the world.  Yes, I can boast that someone has probably been able to dial 911 because I did my job, but I’ve never met anyone who’s used a Motorola Q9h…and I sincerely doubt anyone outside of me and my old HP team have touched the Compaq Airlife.

So, there’s always been something a little bit anticlimactic to much of my career.  Today has been something to change that narrative, though.  In those demos, you actually saw my work.  You don’t know that you saw it, but you did.  Things in those demos worked because I took care of them.  That itself is enough for me.  In this moment, I feel that I’ve arrived, and I have a great deal of excitement about the potential future of this product.  It’s taken several years of paying my dues, but I can finally point to something in the tech world and say that it’s me in there.  Deep down inside that device, there’s a whisper from the hopeful mind of a little boy in Garfield pajamas staccato-typing in BASIC and dreaming of Silicon Valley.

Just keep that in mind if you happen to be shopping for a tablet this summer.

The Bootloader Two Step (aka The x86 Blues)

I knew it would happen early in the project, but I didn’t know it would happen this early in the project.  Incidentally, as I work on the lowest-level parts of ArcanOS, I become increasingly amazed that x86 became the dominant operating system that it did.

The short version for those of you who don’t want to know the gory details– basically your PC is still capable of booting MS-DOS 3.3 off a floppy if you needed it to.  Think about a PC circa 1990 or so.  Think about one today.  Note the difference.  Yet your PC still carries around compliance with standards for that PC back in 1990.  Probably earlier.  Turning it into a modern computer requires enabling and disabling various hardware features in a process that resembles tap dancing on an icy pond in hell.  I’ve just stepped on another crack in the ice, and I’m nerd-raging about it on my blog.  There.  Enjoy your coffee.

So, what could I possibly be talking about?  Well, I’ve been putting work in on the real memory manager for ArcanOS.  Everything in ArcanOS FOOL is basically some Mickey Mouse stuff to just get it booting.  Now I’m sitting down to do it for real, part by part, starting with memory management.  The first step, as you might imagine, is to find out how much physical RAM you actually have so that you know where you can start placing some large memory management data structures like the page table and page directory.  So, how do you find out about the physical RAM?  It’s not like it’s stuffed in a single block of memory addresses.  No.  It’s in bits and pieces all over the place, and to get a reliable map of where it all is, you ask the BIOS to provide it.  So, that shouldn’t be a problem, right?

Well, it is.  Why?  Because the BIOS is available only when you’re in 16-bit “real” mode, and ArcanOS, like most operating systems, uses 32-bit “protected” mode, and one of the first things the bootloader does is get into 32-bit mode so that the OS can get on with its life.  So, after locating some sample code for querying for the memory map, I put it in the bootloader before the switch to protected mode.  “No problem,” I think.  “I’ll stuff the memory map in a guaranteed region of memory and it’ll be waiting for me when the kernel loads.”  I find some sample code, convert it to AT&T syntax, and jam it into the bootloader.  Easy peasy, right?

Nope!  And, why would that be, Bobby?  Again, your PC, at boot time, is stuck in the 1980s.  Booting is a fairly straightforward process for your PC.  It basically looks at your hard drive and loads the first 512-byte sector off of it into a spot in memory and lets it run.  It’s very important that I note that this means the bootloader must be smaller than 512 bytes (and some of those bytes have to be special signatures to prove it’s actually bootable).  ArcanOS FOOL can enable high memory, switch to protected mode, and load and execute the ArcanOS kernel in just under 512 bytes.  Adding the code to get the memory map threw it over its quota by about 20 bytes.

So, where does this leave me?  Really, it leaves me the same place as most people who’ve walked down this road.  I have to make a two-stage boot loader.  Basically, this means that ArcanOS grows from two independent programs (bootloader and kernel) to three.  The first is a bootloader that does nothing but load the second bootloader.  That bootloader can be as big as it wants because it won’t be subject to the single sector limitation.  The first stage loads the second one, and the second one does all the real work.  This does change the ArcanOS build process as well as its disk layout.  In reality, I should now spend those 512 bytes doing something useful like reading the hard drive partition table and locating the second stage bootloader that way.  Ultimately, this will form the rudiments of the ArcanOS file system, since the second stage bootloader is often stashed at a region set aside by the file system.  That means I have to start making tools to provide the filesystem structure and disk layout, whereas the hard disk image for ArcanOS right now is something closer to a floppy.

All because I wanted to know how much memory the computer has and where the memory is kept.  Again, this is the dominant hardware architecture of PCs, laptops, and netbooks.

There is an alternative which I’m considering, and that’s to learn to use GRUB.  GRUB is designed to provide flexible and rich boot features for any OS project.  GRUB will cheerfully assemble a memory map for me, it’ll make pretty boot menus, and I think it might also correctly foam a latte.  And, in fact, GRUB compatibility is absolutely on my mental list for ArcanOS PRIESTESS but I’m doing my best to keep it out of ArcanOS MAGUS.  Why?  Because the primary goal of ArcanOS is to teach.  I need to go through the challenge of a two-stage boot loader and some physical memory detection first, and then, once I know what’s really going on down there, I’ll bring ArcanOS up to GRUB multiboot compliance and make use of its fun extra features.

This also, incidentally, demonstrates why there is a gap between the “Hello World!” bootloader tutorial and the “How to use GRUB” tutorial.  It’s because doing anything meaningful with a bootloader requires a two-stage bootloader and is complex and headache-inducing…definitely not the sort of thing that’s quite so easily explained in a quick blog article.  Hopefully, this audit trail of my struggles will help improve that, even though I will be doing like a good open source coder and, after ArcanOS MAGUS, leave behind some of the irritations of x86 hardware and boot and let GRUB do the heavy lifting.

ArcanOS FOOL released

It may have taken a little longer than I would have initially liked, but I’m pleased to announce that, as of this evening, ArcanOS has had its first milestone release — FOOL.  As I have mentioned in the ArcanOS wiki on Google Code, I think version numbers are pretty arbitrary, so I have instead opted to make milestone releases and name them after the trump cards in a tarot deck.  The goals for FOOL are pretty apt for its name.  In a tarot deck, “the fool” is often depicted as a young man blissfully stepping off a cliff.  The goals for this release were to basically lay the groundwork for getting to the “real work.”  This meant having a coherent build process and test/debugging platform and developing a bootloader, rudimentary kernel, basic debugging tools (like logging), exception handlers, and at least one interrupt handler.  Having all these basic pieces in place means that I have something stable I can use for future development.

Currently, ArcanOS is little more than some initialization code and a spinlock.  There is a basic handler for the keyboard interrupt, but that’s it.  There’s very little to try out, as there’s no user-space and thus no programs written for it.  Still, I’m pretty pleased with it.  It’s the first personal project I’ve taken on since 2001, and most personal open-source projects don’t even make it to their first milestone release.

The next release will be ArcanOS MAGUS.  I’m currently scoping out my goals for it.  I know I want to move the memory model to a flat, paged model.  This will affect multiple parts of the code base because it’ll mean that I will be able to stop adjusting for the segmentation offset whenever I need a linear or physical address.  I suspect this alone will make paging the major effort, but I want to do some other things that will keep moving things forward.  After all, having an environment for user processes is the point at which things get really fun.

As I mention in the ArcanOS notes, I owe a debt of gratitude to Dr. Lorenzo Alvisi as well as the xv6 project over at MIT.  xv6, in an earlier iteration, was known as JOS, which Dr. Alvisi used to produce a series of operating systems lab exercises for his operating system courses.  ArcanOS started out as one of those lab exercises, and though it has changed very significantly already, it still bears much of the original bootloader code.