The Bootloader Two Step (aka The x86 Blues)

I knew it would happen early in the project, but I didn’t know it would happen this early in the project.  Incidentally, as I work on the lowest-level parts of ArcanOS, I become increasingly amazed that x86 became the dominant operating system that it did.

The short version for those of you who don’t want to know the gory details– basically your PC is still capable of booting MS-DOS 3.3 off a floppy if you needed it to.  Think about a PC circa 1990 or so.  Think about one today.  Note the difference.  Yet your PC still carries around compliance with standards for that PC back in 1990.  Probably earlier.  Turning it into a modern computer requires enabling and disabling various hardware features in a process that resembles tap dancing on an icy pond in hell.  I’ve just stepped on another crack in the ice, and I’m nerd-raging about it on my blog.  There.  Enjoy your coffee.

So, what could I possibly be talking about?  Well, I’ve been putting work in on the real memory manager for ArcanOS.  Everything in ArcanOS FOOL is basically some Mickey Mouse stuff to just get it booting.  Now I’m sitting down to do it for real, part by part, starting with memory management.  The first step, as you might imagine, is to find out how much physical RAM you actually have so that you know where you can start placing some large memory management data structures like the page table and page directory.  So, how do you find out about the physical RAM?  It’s not like it’s stuffed in a single block of memory addresses.  No.  It’s in bits and pieces all over the place, and to get a reliable map of where it all is, you ask the BIOS to provide it.  So, that shouldn’t be a problem, right?

Well, it is.  Why?  Because the BIOS is available only when you’re in 16-bit “real” mode, and ArcanOS, like most operating systems, uses 32-bit “protected” mode, and one of the first things the bootloader does is get into 32-bit mode so that the OS can get on with its life.  So, after locating some sample code for querying for the memory map, I put it in the bootloader before the switch to protected mode.  “No problem,” I think.  “I’ll stuff the memory map in a guaranteed region of memory and it’ll be waiting for me when the kernel loads.”  I find some sample code, convert it to AT&T syntax, and jam it into the bootloader.  Easy peasy, right?

Nope!  And, why would that be, Bobby?  Again, your PC, at boot time, is stuck in the 1980s.  Booting is a fairly straightforward process for your PC.  It basically looks at your hard drive and loads the first 512-byte sector off of it into a spot in memory and lets it run.  It’s very important that I note that this means the bootloader must be smaller than 512 bytes (and some of those bytes have to be special signatures to prove it’s actually bootable).  ArcanOS FOOL can enable high memory, switch to protected mode, and load and execute the ArcanOS kernel in just under 512 bytes.  Adding the code to get the memory map threw it over its quota by about 20 bytes.

So, where does this leave me?  Really, it leaves me the same place as most people who’ve walked down this road.  I have to make a two-stage boot loader.  Basically, this means that ArcanOS grows from two independent programs (bootloader and kernel) to three.  The first is a bootloader that does nothing but load the second bootloader.  That bootloader can be as big as it wants because it won’t be subject to the single sector limitation.  The first stage loads the second one, and the second one does all the real work.  This does change the ArcanOS build process as well as its disk layout.  In reality, I should now spend those 512 bytes doing something useful like reading the hard drive partition table and locating the second stage bootloader that way.  Ultimately, this will form the rudiments of the ArcanOS file system, since the second stage bootloader is often stashed at a region set aside by the file system.  That means I have to start making tools to provide the filesystem structure and disk layout, whereas the hard disk image for ArcanOS right now is something closer to a floppy.

All because I wanted to know how much memory the computer has and where the memory is kept.  Again, this is the dominant hardware architecture of PCs, laptops, and netbooks.

There is an alternative which I’m considering, and that’s to learn to use GRUB.  GRUB is designed to provide flexible and rich boot features for any OS project.  GRUB will cheerfully assemble a memory map for me, it’ll make pretty boot menus, and I think it might also correctly foam a latte.  And, in fact, GRUB compatibility is absolutely on my mental list for ArcanOS PRIESTESS but I’m doing my best to keep it out of ArcanOS MAGUS.  Why?  Because the primary goal of ArcanOS is to teach.  I need to go through the challenge of a two-stage boot loader and some physical memory detection first, and then, once I know what’s really going on down there, I’ll bring ArcanOS up to GRUB multiboot compliance and make use of its fun extra features.

This also, incidentally, demonstrates why there is a gap between the “Hello World!” bootloader tutorial and the “How to use GRUB” tutorial.  It’s because doing anything meaningful with a bootloader requires a two-stage bootloader and is complex and headache-inducing…definitely not the sort of thing that’s quite so easily explained in a quick blog article.  Hopefully, this audit trail of my struggles will help improve that, even though I will be doing like a good open source coder and, after ArcanOS MAGUS, leave behind some of the irritations of x86 hardware and boot and let GRUB do the heavy lifting.

Morning Surf at Pleasure Point

It’s been hot the last couple of days.  When I say that it’s been hot, I mean that it’s been hot even for a Florida boy like myself.  On top of that, California’s heat is very different from Florida’s.  Heat in Florida is almost always wet.  The air is thick enough to smack you in the face, and you can smell sauna-like moisture everywhere.  When the sun beats down like God’s flashlight shining directly on your pathetic life, the heat and the humidity trigger natural instincts to stop being stupid and go find some shade and air conditioning.  California heat, by comparison, is very dry and the sun is far more indirect.  It’s a sort of deceptive heat, one that makes you wonder when, exactly, you stepped into a blast furnace.  In my experience, it can sneak up on you and give you exposure almost before you realize it’s happened.

It’s been a few weeks since I went surfing, too.  What better way to beat the heat and turn down the stress factor of a workday in Silicon Valley than to go surfing at dawn?

So, I got up this morning and did just that.  I was a little late getting out the door, but not too bad.  Cowell’s Beach has been hit-and-miss (mostly miss) lately, so I decided to expand my experience in the greater Santa Cruz area and went to Pleasure Point.  It’s a truly beautiful surf spot, though I’m not completely sure how productive the trip was.  Basically, I didn’t get any actual wave riding done.  I’m not really bothered by that, though, because I’m still very much a beginner and I did get to work on plenty of my basic skills like paddling and sitting on my board.  My chest and shoulders are deeply fatigued now.  It’s clear I need to add upper body exercise to my weekly schedule!  The other major reason that I’m not bothered by not getting any riding done is that I wasn’t the only one with that problem.  The swells that were coming in kept breaking at different points.  I started pretty far out (wanting the workout of paddling out) and spent a lot of time staring at waves breaking another 100 feet in.  Content to watch the other surfers and learn a thing two, I stayed out.  When I (and a couple of others who were far out) moved in, though, the waves started breaking farther out, and I spent a lot of time fighting whitewash.  Sadly, I didn’t have all morning to just keep waiting and, after an hour, I was actually pretty tired, so I paddled in and packed up.

It was fun, though, popping out in the early hours of the morning to spend time in the ocean and get a workout very different from my usual routine of cycling twice daily.  I took my own trip out with my own gear, which was a first, and I made a friendly impression on the other surfers out there.  Given that this is a sport where safety and basic control come first, I consider it a good use of my time, even if I didn’t get to work on my pop-up outside of my living room.