lunes, 21 de marzo de 2011

SqueakNOS release updated!

We are glad to announce that we have just uploaded an updated version of SqueakNOS to our site. You can find the updated download links here.

This release adds the following items to the previous version: (99.9% modeled with objects)

- Improved support for ATA devices access
- Implementation of a model of files and directories
- Implementation of FAT32 filesystem
- Support for reading sources from the hard disk
- Support for X86 paging setup and handling of page faults
- Implementation of a model of free and used memory
- Implementation of image snapshotting (based on the paging mechanism)
- Based on Pharo, but also supporting Squeak and Cuis
- Added some primitives for communicating between VM and image
- Support for Alien and FFI
- Added basic support for writing to serial via VM so that it's possiblo to out debug info to the outside world.
- Added pretty basic ATA benchmarks
- Added some scripts to automatize work done from the outside world
- Many code cleanups

Future directions are:

- Support for Cog VM
- Investigate replacing SqueakNOS plugin with NativeBoost so that we do more on the image
- Offloading memory pages to disk
- More tests
- Benchmarks

For the brave ones, remember that you can always generate a SqueakNOS image by using the ConfigurationOfSqueakNOS (which is a bit outdated now), or by manually loading the packages from monticello. Also to generate a SqueakNOS VM download its code from GIT (hosted on sourceforge.net).

Don't forget to visit squeaknos.blogspot.com for news and instructions. We hope to be able to write more soon. Have fun!

Many thank's t all the people who made it possible to understand the code and to develop all this.

SqueakNOS' Team

jueves, 6 de mayo de 2010

Building new releases of SqueakNOS

We roll new versions of SqueakNOS periodically by using a really simple automated process, and this article explains how it works. So, after reading this, you'll know how to rebuild SqueakNOS Object Engine, create a new bootable hard disk or ISO image, and roll your own version. This process should be as easy and fast as possible.

Quickly rebuilding and trying

We work on Linux, so it will be a lot easier to do everything from Linux. However, the same should work on other Unixes, and quite probably in Windows too, after finding the right versions of the tools.

The distribution package of SqueakNOS has changed a bit since May 2010. If you look at packages from before that date, directories may be a bit different. Now we include three things in the package: the compiled bootable system in a hard disk or ISO images, a .vmx configuration file to boot the image from vmware and lastly the native source code of the Object Engine.

To build everything you have to unpack the source files into your filesystem, which lie in "platforms" directory. This contains a subset of the subversion tree of the standard Squeak build tree (the cross platform part), and an added SqueakNOS target platform. Everything is done from within platforms/squeaknos, where the main Makefile is located.

$ tar -xzf SqueakNOS.tar.gz .
$ cd platforms/squeaknos

Then copy, from the hard disk or ISO image the two files SqueakNOS.image and SqueakNOS.changes:

$ # To mount the ISO image
$ mount -o loop SqueakNOS.iso /nos

$ # Or to mount the hard disk image
$ vmware-mount SqueakNOS.vmdk /nos

$ cp /nos/SqueakNOS.image .
$ cp /nos/SqueakNOS.changes .

Now you are ready to roll your own SqueakNOS.iso:

$ # To simply build the kernel, obviously:
$ make

$ # To build release/SqueakNOS.iso and release/SqueakNOS.vmx:
$ make iso

$ # To build and try it with vmplayer:
$ make try

$ # To try with qemu
$ make iso && qemu -std-vga -cdrom release/SqueakNOS.iso

$ # To build a new distro (release/SqueakNOS-`date`.zip and tar.bz2):
$ make distro    # The date in the filename is automatic

Of course, you can always open the .image in a standard squeak, modify it, save it, and try it. The .image and .changes are copied to release/iso during the building of the .iso image, so you can continue working inside squeak while building and trying. This is great to make changes while you are debugging in a live SqueakNOS. It's currently also possible to have access to .changes and .sources from within SqueakNOS itself using while using the hard disk image or using a RemoteObjectChannel over a virtual serial line with the HD/ISO images, if you are interested in doing this, take a look at this article.

jueves, 29 de abril de 2010

Accessing Squeak's image from the outside

   After some time playing with SqueakNOS, you'll notice you can't save the image from the inside of the VM. Also, you may want to introduce some code from the outside,  and if you are like me, you just like working in normal squeak and then migrate the changes to the VM. So there has to be a way of modifying the image.
   To do so, you have to mount the Hard disk/ISO image in a folder, read/write the .image and .changes files and then unmount.

Mounting the virtual hard drive

In case you are using the VM appliance, you'll mount the VM disk's file system inside your real file system. You can do that with this command (which gets installed with VMWare Player)
$ vmware-mount SqueakNOS.vmdk /vmdisk
You have read/write access here, so you can work with normal squeak, save the image and finally copy it overwriting SqueakNOS.* inside the mounted directory. When you are done you do
$ vmware-mount -k ../SqueakNOS.vmdk
to unmount. When you restart the VM you'll see all the changes you did from the outside.

Accessing the ISO image

Here we get into trouble: ISO images can only be mounted as read-only. This means that you'll be able to read the files inside it, but wouldn't be able to write to it a new image in case you make a change from the outside. To solve this problem, you can regenerate the ISO with a new image.

Unpack the .iso in some place, overwrite the .squeakNOS.image/changes files and then issue:
~/isocontent/ $ cd platforms/SqueakNOS
~/isocontent/platforms/SqueakNOS$ make iso
that will generate in release folder a new bootable .iso with the image you just placed.

Getting started with SqueakNOS

The idea of this article is to explain what are the different bundles that are available to deliver SqueakNOS.
You probably already know that SqueakNOS is(n't) an operating system. Then, the way to use it is by booting it's code from some media, like a CD or a hard drive. The good news is that it's not necessary to alter your hard drive in any way, nor reset your PC each time you want to test SqueakNOS, and that's possible thanks to virtualization. The idea of virtualization is to simulate the existence of an entire machine (processor, memory, hard disk and other devices), where you can install an entire operating system. The software used to do this is called a virtualizer, and there are many of them available, including VMWare Player, Sun's VirtualBox, Bochs, QEmu, etc.

SqueakNOS bundles


SqueakNOS comes bundled ready to be used in two ways:
  • as a bootable CD, actually an ISO image, which you can burn and boot in your machine or load into the virtualizer to simulate a CD (as you'll see in the next section).
  • as a small bootable virtual hard drive, prepared to run inside the virtualizer.
It isn't necesary to download them both, you should choose the one you feel more comfortable and use it. The virtual harddisk has the benefit of allowing you to mount it read/write in your filesystem if you need. With the ISO you can mount it read-only, and if you want to change any file you have to regenerate the image (which gets done with one simple command).

To download any of the bundles go to SqueakNOS files section in SourceForge.
As of 19-feb-2010, the hard disk image is some days more recent than the ISO image, but that should change quickly, we are preparing an update right now.

SqueakNOS in a virtualized environment


The easier way to test and work with SqueakNOS is through virtualization. That means that with the help of a virtualizer you'll create a virtual machine where you can install and boot the SqueakNOS easily. I use VMWare Player because it's a the fastest virtualizer I know, and also because it comes with vmware-mount tool which is useful to access disk images outside the VM. But there are many others supported, including Sun's VirtualBox, Bochs, QEmu, Microsoft's VirtualPC and Parallels.
Both ISO and hard drive images come with a .vmx file. This is a simple text file that VMWare uses to describe the virtual hardware. By opening it with VMWare Player you can run SqueakNOS. It takes just seconds to load the image, after this you should see the video mode changing.

SqueakNOS in a real system


The easier way of trying SqueakNOS in a real system would be to burn the bootable ISO to a CD and boot from it. Installing it to the hard drive is a bit more complex and will be addressed in another article.

miércoles, 21 de abril de 2010

Phase 3: Give it awareness [Almost done]

Make keyboard and mouse work. This let us actually try SqueakNOS. Make hard drives work, this will let us code SqueakNOS from within SqueakNOS.
Having keyboard and mouse opened the door to playing with the booted system. Now, we want to be able to save the changes, so we need some persistency. One idea (taken, again, from the old SqueakNOS), was to implement serial port support, which is already done, and some type of File System over it.

Other possibility is to implement hard disk drivers, which is also done, and after that basic FAT32 support. This will let us open .sources and .changes files.

Phase 2: Make it breath [Done]

Add graphics mode support, and native IRQ setup and handling
Graphics mode

After the nice suprise that GRUB had graphic mode switching almost ready to be used, we jumped into making it work for us, and we chose a graphic mode that is really nice AND maps 1 to 1 from Squeak's representation to video RAM: 1024x768x32 bits is our choice although any 32 bits mode would suit the needs. Hacking a ioShowDisplay() that works was easy: it's no more than a bounded memcpy(). With this we got some graphic output in SqueakNOS' screen!

With the old SqueakNOS we had some problems at this point, mainly because Squeak does not refresh the screen upon entering, so this time we solved it really fast, by doing a "self currentWorld fullRepaintNeeded" after a small Delay, and voila! Squeak's full screen was shown!

Now we were facing the most delicate part: making IRQs properly work. Subtasks: set a void handler for every IRQ and enable IRQs (see that it doesn't halt). Add an infinite loop handler for the keyboard IRQ (see that it hangs on a keypress). Set a visible handler for the timer interrupt (see that it draws something on the screen... you can still see it in the upper right corner of SqueakNOS). Finally change all void handlers for code that will trigger a Squeak Semaphore and exit, also add the primitive to register the Semaphores (see that it doesn't hang, then see that the Semaphore gets triggered from inside Squeak). At last, loop waiting on the semaphore for keyboard's IRQ, read keyboard's port, and signla IRQ end all from inside Squeak. [Done]

SqueakNOS IRQ Handling

Squeak VM lets you signal Squeak Semaphores from the native world by calling signalSemaphoreWithIndex(). We want to serve IRQs from Squeak, using interpreted code, not native code. We seriously think that with hardware close to 1000 times faster than 20 years go we should be able to do it without any problems. Of course we cannot set the native IRQs handlers to jump to Squeak code, so our idea is to have a different Semaphore for every IRQ and have a Squeak Process with highIOPriority blocking on the Semaphore.

The code inside Squeak, taken from InterruptRequestDispatcher>>installOn: looks like:
installOn: aComputer
self registerSemaphore.
process := [
[
semaphore wait.
self handleOn: aComputer.
aComputer interruptController signalEndOfInterrupt: interruptNumber.
] repeat
] fork priority: Processor highIOPriority.
aComputer interruptController enableIRQ: interruptNumber

And the native code to signal the Semaphore, taken from ints.h, is similar to:
 void irq_1_handler();
asmlinkage void ISR_1() {
if (0!=IRQSemaphores[number])
signalSemaphoreWithIndex(IRQSemaphores[number]);
}

asm(
".text"
".align 16"
"irq_1_handler:"
"pusha"
"call ISR_1"
"popa"
"iret"
)

There is an interesting detail in all this: the software interrupt ending (IRET) and the hardware IRQ ending (outb(0x20,0x20)) are detached in SqueakNOS, were they are almost always done at the same time in every other code we saw. This gives a really desirable result: The IRET lets the software continue, going back to the interpreter and letting the Process waiting on the Semaphore be rescheduled, however, the hardware part (Interrupt Controller) still thinks the IRQ has not been served yet, and will wait until the Squeak side of the handler (shown above) signals the end of the interrupt (aComputer interruptController signalEndOfInterrupt: interruptNumber.

We seriously believe that with computers close to 1000 times faster than 20 years ago this should be ok, however we are not sure yet if it may bring any problems later, and we are open, although reluctant, to the possibility of coding some glue code, or even complete "device drivers" natively (or hopefully using Exupery). Probably for sound and video I/O, we'll see.

We were successfully using the very same scheme in the old SqueakNOS, so we just ported it to the new code base.

Phase 1: Make it boot [Done]

Make it boot and start interpreting the .image with text output for tracing and debugging.


The old SqueakNOS used a custom made mutant boot loader, borned from mixing LILO, the Linux kernel boot loader, and our dirty fingers. We had quite a few problems with it, and it was really complicated to put the needed environment to compile it. So we chose to replace it for something better and more modern: GRUB.

We first thought on GRUB because it, of course, handles all initialization, switches to flat 4G mode can load huge kernles in high memory, and also has support for modules loading (module and modulenounzip GRUB commands). We wanted to detach the native part and the Squeak .image, which were glued in a single file in the old SqueakNOS, making it easier for Squeakers to touch the .image without having to set up the compiling environment.

We needed to implement the switch to graphics mode before starting jumping to the interprenter. In the old SqueakNOS we did it from real mode (that's why we had so poor video support). But this time GRUB gave us control already in protected mode, so we couldn't just call the right INT. Luckily, GRUB already has the roots for supporting graphic video mode selection, and although it is not usable in the standard GRUB distro, we could solve the problem quite easily, by copying the testvbe command into the setvbe command, and removing everything after the command switches the video mode. We later found out that the video memory address changes from box to box, so we also hacked the vbeprobe command to show the video memory address.

We also learned that GRUB supports part of the multiboot protocol and is able to load ELF binaries, this led us reduce the amount of assembly code to barely around 20 line. It's not that we are afraid of assembly, but if multiboot was supported in some other platform (Apple/PowerPC), porting SqueakNOS' kernel should be really easy. [Never tried]

At this point we had GRUB loading the kernel and loading the .image all in memory, we had some glue code to make the .image look like a file so Squeak's intepreter could start interpreting it. We coded console text output support, hacked some debugging messages in the interpreter loop, some stack traces here and there and figured out what Plugins needed to be compiled for Squeak to start up: we finally managed to make the interpreter loop run loose.