Apache 2.4.10 Abort (core dumped) on SmartOS 20140919T024804Z


Just a few hours after posting this, Joyent committed the fix to the Illumos kernel on github.  Kindly, Mike Zeller informed me of that fact on Twitter.  I love the internet and open source software.


Original post:

Recently, I upgraded my home SmartOS server to 20140919T024804Z.  Generally this is a simple process where I download the latest USB image from smartos.org, image it to a 2GB USB stick, and boot from it.  This process usually takes no more than 10 minutes to complete.

This time, however, was a bit different.  I wasn’t able to reach the apache server on one of my zones (megaman). Running svcs -xv quickly told me that yes, apache was not running.

[root@megaman ~]# svcs -xv
svc:/pkgsrc/apache:default (Apache 2 HTTP server)
State: maintenance since September 27, 2014 11:22:26 PM UTC
Reason: Start method failed repeatedly, last dumped core on Abort (6).
See: http://illumos.org/msg/SMF-8000-KS
See: man -M /opt/local/apache2/man -s 8 httpd
See: http://httpd.apache.org
See: /var/svc/log/pkgsrc-apache:default.log
Impact: This service is not running.

Tailing the service log, I found that the httpd binary was returning an ABRT signal.

[root@megaman ~]# tail /var/svc/log/pkgsrc-apache:default.log
[ Sep 27 23:14:25 Method "start" failed due to signal ABRT. ]
[ Sep 27 23:14:25 Executing start method ("/opt/local/sbin/httpd -k start"). ]
[ Sep 27 23:14:25 Method "start" failed due to signal ABRT. ]
[ Sep 27 23:22:24 Enabled. ]
[ Sep 27 23:22:26 Executing start method ("/opt/local/sbin/httpd -k start"). ]
[ Sep 27 23:22:26 Method "start" failed due to signal ABRT. ]
[ Sep 27 23:22:26 Executing start method ("/opt/local/sbin/httpd -k start"). ]
[ Sep 27 23:22:26 Method "start" failed due to signal ABRT. ]
[ Sep 27 23:22:26 Executing start method ("/opt/local/sbin/httpd -k start"). ]
[ Sep 27 23:22:26 Method "start" failed due to signal ABRT.

Attempting to run the httpd binary manually aborted almost immediately:

[root@megaman ~]# cd /opt/local/sbin/
[root@megaman /opt/local/sbin]# ./httpd
Abort (core dumped)

This was very strange considering apache has worked perfectly in megaman for years. At this point I had only updated SmartOS but none of the pkgsrc packages on the zone. My guess was that something was amiss between the application on the zone and the new system libraries. I used pstack to glance that the process stack of the core file.

[root@megaman /opt/local/sbin]# pstack core
core 'core' of 37746: ./httpd
fffffd7fff1fbd3a _lwp_kill () + a
fffffd7fff191160 raise (6) + 20
fffffd7ffec47e14 umem_do_abort () + 44
fffffd7ffec48065 ???????? ()
fffffd7ffec4ad80 process_free (59c810, 1, 0) + a0
fffffd7ffec4aecd umem_malloc_free (59c810) + 1d
fffffd7ffed48425 __IPv6_cleanup (5867c0) + 25
fffffd7ffed46e35 getipnodebyname (fffffd7fffdff420, 1a, 8007, fffffd7fffdff53c) + 115
fffffd7ffea597ea get_addr (0, 53e9e8, fffffd7fffdff5f0, fffffd7fffdff5f0, 0, 0) + 10a
fffffd7ffea59044 _getaddrinfo (53e9e8, 0, fffffd7fffdff700, fffffd7fffdff6f8, 0) + 2a4
fffffd7ffea596a0 getaddrinfo (53e9e8, 0, fffffd7fffdff700, fffffd7fffdff6f8) + 20
fffffd7ffb4dc7b5 call_resolver.isra.0 () + 65
0000000000467e9f ap_set_listener () + 1bf
000000000045cf3d invoke_cmd () + 61d
000000000045f37f ap_walk_config () + af
0000000000460560 ap_process_config_tree () + c0
0000000000477d16 main () + 6a6
000000000043ef2c _start () + 6c

getipnodebyname appeared to be causing the ABRT of the httpd process.  I  decided to capture parameter information for getipnodebyname to determine what caused it to go haywire.  MDB to the rescue!

[root@megaman /opt/local/sbin]# mdb ./httpd
> ::run
mdb: stop on SIGABRT
mdb: target stopped at:
libc.so.1`_lwp_kill+0xa:jae +0xc <libc.so.1`_lwp_kill+0x18> mdb: You've got symbols!
Loading modules: [ ld.so.1 libumem.so.1 libc.so.1 ]

Cool, ABRT in mdb too. Now on to the interesting part. Run the ::stack dcmd and find the hostname that was sent to getipnodebyname.

> ::stack
libumem.so.1`process_free+0xa0(59e810, 1, 0)
libnsl.so.1`getipnodebyname+0x115(fffffd7fffdff420, 1a, 8007, fffffd7fffdff53c)
libsocket.so.1`get_addr+0x10a(0, 5409e8, fffffd7fffdff5f0, fffffd7fffdff5f0, 0, 0)
libsocket.so.1`_getaddrinfo+0x2a4(5409e8, 0, fffffd7fffdff700, fffffd7fffdff6f8, 0)
libsocket.so.1`getaddrinfo+0x20(5409e8, 0, fffffd7fffdff700, fffffd7fffdff6f8)

> fffffd7fffdff420::dump
\/ 1 2 3 4 5 6 7 8 9 a b c d e f v123456789abcdef
fffffd7fffdff420: 3a3a00ff 7ffdffff 3eb33bff 7ffdffff ::……>.;…..

See the first two bytes of the string? In hexidecimal it’s 3a3a.  getipnodebyname appears to be failing while trying to look up “::”.  A double colon is shorthand in IPv6 that means the equivolent of in IPv4.

My next thought was to disable the IPv6 interface to see if that made any difference. First run ipadm show-addr to find any IPv6 addresses.

[root@megaman /opt/local/sbin]# ipadm show-addr
lo0/_a static ok
net0/_a from-gz ok
lo0/_b static ok ::1/64

Ah, an IPv6 localhost entry.  Disable the IPv6 interface lo0/_b.

[root@megaman /opt/local/sbin]# ipadm disable-addr -t lo0/_b

Test apache again.

[root@megaman /opt/local/sbin]# ./httpd

It works!  No abort and the daemon started without trouble.  svcadm enable apache also worked without a problem.

Obviously, disabling IPv6 is decidedly not a permanent solution.  A few months from now if httpd dies again, I don’t want to be scratching my head trying to remember what could be causing it.  This is a good example of Technical Debt.  Trading long term difficulties for short term “just getting it to work”.  Best avoided.

Luckily, this httpd daemon is running a simple web application that’s trivial to migrate to a new smart machine.  After a simple imgadm import and vmadm create, I had a brand new zone.  It just took a few minutes to migrate the application to it (now with LZ4 compression!).

Posted in computers, solaris | Leave a comment

Install ZFS on Debian Testing (Jessie)

ZFS on Linux

ZFS on Linux

I’ve been using ZFS on Linux  to provide ZFS functionality on Linux for sometime now.  I first became attached to the feature set of ZFS while maintaining Sun Solaris systems a few years ago.  I mostly manage a variety of Linux systems now and have found ZFS on Linux to be stable enough for my development work.  Though, many folks are using it in production very successfully!

As a matter of fact, Jim Salter wrote the following on Ars Technica recently referencing ZFS on Linux:

I’ve been using ZFS professionally and in production for over five years, and I can honestly say that it’s both changed the course of my career and my business. I wouldn’t dream of going back to the way I did things before ZFS.

That’s quite a ringing endorsement but Jim has many more things to say about ZFS and its strong and weak points.  I found myself nodding my head throughout most of it.  The article is well worth a read.

As an aside, you can watch the maintainer of ZFS on Linux, Brian Belendorf of LLNL, give a talk at Joyent’s ZFS Day on his motivations to write ZFS on Linux.  His talk begins at 31:40 here.

Since my Linux distro of choice for development boxes is Debian testing (Jessie), I hoped the installation for Debian stable (Wheezy) described on this page would work.  Unfortunately, there is an issue with ZFS on Linux version 0.6.2 running kernels greater than 3.12.  Basically, a new API was introduced to the Linux kernel that is incompatible with ZFS on Linux.  The incompatibility has been fixed, but the official ZFS on Linux release hasn’t been updated.

Unfortunately, this bug prevents ZFS on Linux from compiling and installing the kernel module on Debian testing.  I have tested the  instructions on Debian stable and found it works perfectly.  The following steps are only necessary for folks who want to run Debian testing (or above).

I decided to take the leap and run bleeding edge ZFS on Linux on my Debian testing systems.  Note that this is not for the feint of heart since this code has not be throughly tested and there are likely bugs.  I would not run it in production.  So, caveat emptor and all that.

First, there are several dependencies to get out of the way.  ZFS on Linux requires libz, uuid, and the Linux kernel headers.  These dependencies can be taken care of with a simple aptitude installation.  The following command will do the trick.

sudo aptitude install git autoconf libtool zlib1g-dev uuid-dev linux-headers-$(uname -r)

Now create a directory to download and compile ZFS on Linux.

mkdir ${HOME}/zfs
cd ${HOME}/zfs

Time to download the code!  You’ll need the Solaris Porting Layer (SPL) and ZFS git repositories.  You can use ‘git clone’ to download both repos.

git clone https://github.com/zfsonlinux/spl.git
git clone https://github.com/zfsonlinux/zfs.git

Downloading the two repositories should create two directories names ${HOME}/zfs/spl and ${HOME}/zfs/zfs.    First, you’ll need to compile and install the SPL module.

cd ${HOME}/spl
./configure --prefix=/usr
sudo make install

Now compile and install the ZFS module.

cd ${HOME}/zfs
./configure --with-spl=${HOME}/zfs/spl --prefix=/usr
sudo make install

Next you should be able to start your newly compiled ZFS kernel module.

sudo modprobe zfs

If the module has loaded properly, you should be able to run ZFS management utilities such as zfs and spool.  On Linux, you must run zfs and zpool as root.  You’ll get this error if you don’t.

jgrafton@debian-testing:~$ /usr/sbin/zfs list
Unable to open /dev/zfs: Permission denied.

Once zpool and zfs are working properly, you’ll be able to create ZFS datasets.

jgrafton@debian-testing:~$ sudo zpool create -f testing /dev/sdb
jgrafton@debian-testing:~$ sudo zpool status
  pool: testing
 state: ONLINE
  scan: none requested

	testing     ONLINE       0     0     0
	  sdb       ONLINE       0     0     0

errors: No known data errors

This procedure has worked for me for awhile now and I haven’t suffered any data loss events.  Again, it’s always a possibility due to the bleeding edge nature of this code.


Posted in computers, zfs | Comments Off

A Git User in a SVN World

I’ve been using git as a part of my development workflow for several years now.  The past ten years or so have been a slow progression from subversion to mercurial to git.  I really like all the features that git has to offer and the ease of branching has made my development life much easier.

Every once and awhile, however, I run into a situation where I need to use subversion.  Usually this is because someone else’s code means someone else’s repository of record.

I recently found myself in a situation where asking the organization to migrate from subversion to git just wasn’t feasible.  My next best option was git-svn.  Git-svn is effectively a subversion to git and back to subversion converter of sorts.  You can maintain your git based workflow (slightly modified) and still commit to the organization’s subversion repository.

Frankly, This. Is. Awesome.  Git-svn is included in nearly all git distro deployments (including the XCode CLI from Apple) and is very easy to use.  First, start off with a clone of the subversion repository using git-svn:

git svn clone http://svn-repo.com/svn/repo

This will clone (checkout in subversion parlance) a subversion repository from a remote site to your localhost.  Using this syntax will clone everything from the remote host exactly as is.  Unfortunately, this doesn’t leave much room for branching and tags.  If your remote site is setup with branch, tag, and trunk directories (i.e. http://svn-repo.com/svn/repo/trunk|branch|tag), you can tell git to use the branch directories to emulate git branches using the -s argument:

git svn clone -s http://svn-repo.com/svn/repo

Now your repository is cloned with branching enabled.  You will not see the branch, tag, and trunk directories as they will be emulated in your git repository.

Now you are able to take advantage of your git workflow as usual (adding and commiting) with just a small difference.  If you’d like to send your commits up to the subversion repository, use the command:

git svn dcommit

Your git repository is now uploading your commits to the subversion repository!  For more information, there is a great tutorial on using git-svn at viget.

Posted in computers | Comments Off

Site Offline

Oops, looks like my blog has been offline since Sunday, August 26th.  It appears that my Amazon EC2 instance went offline shortly after the year free time limit.  The credit card I originally used to create the instance (you need it to create the instance but it doesn’t get charged during the one year trial period) expired.

The credit card situation is now fixed and the site should not just automagically disappear anymore.  Sorry about that.

Posted in computers | Comments Off

Life @ 40 Inches in California

This past Christmas (2011), my son Alex received a Fischer Price digital camera.  At the time, Alex was only 2 years old.  My wife, Sara, and I were preparing for our move to California and had our hands full.  The move was hectic, of course, and unbeknownst to Sara and I, Alex was busy snapping pictures.  Granted, many of them were accidental, but I was surprised to find he had taken nearly a thousand (899) pictures between Christmas 2011 and July 2012.

Going through the pictures this afternoon has been an interesting experience for me since I get a rare opportunity to see a significant life adjustment through the eyes of my young son.  Here are a few shots I think capture everyday aspects of our new Cali life.  As far as I’m aware, Alex took all of these pictures.  There may be some where he cajoled a family member to take a picture but they’re few and far between.  He regularly lets us know this is his camera.  Without further ado…

Our Protagonist, Alex Grafton


Mommy's Computer

Packing, Spiderman MUST go on the plane

On the plane at the airport with Mommy


Grandma flew to California and brought the dogs shortly after Alex and Sara flew in


It must be terribly boring in the car seat


He had the camera on the TV stand and was snapping pictures over and over again. Seriously, there are like 100 of these.


This is me coming home from work.

Uncle Matt went kite flying with us in Mountain View!


Tiny Toddler Feet!


Alex got a few pictures of Sara and I at Golden Gate Park. Here's a great one of Sara!


Walking the dogs. We're in an apartment now and have to walk the dogs twice a day. Alex likes to take his camera along.

Riding in the car with Daddy, a common occurrence.

Riding in the car with Mommy, also a common occurrence.

Busted! Mommy says McDonald's is a sometimes food. However, Alex and I needed to get our McNugget on!

Moral of the blog post, if you have small children, buy them an indestructible digital camera and let them loose.  They may not be into it but if they are, the results are wonderful and surprising!


Posted in california | Comments Off

Barriers to Entry for DTrace Adoption

Yesterday, at dtrace.conf, Jarod Jenson gave a presentation on why he thinks dtrace has not seen greater adoption by system administrators across the spectrum of varied IT departments around the world.  (Jarod starts his presentation at the 40 minute mark.)  At the beginning of his talk, Jarod mentioned he didn’t think dtrace’s syntax was a problem and I largely agree with that.  Like any language syntax, dtrace becomes more familiar the more you use it.  I believe Jarod hinted upon the correct answer a little later in his presentation but in my opinion he missed the mark a bit.

At my last job, I worked as the manager of a team of Solaris administrators for a large public university in Texas.  This was my first exposure to an operating system that was dtrace enabled.  At the beginning of my tenure, the shop ran mostly Solaris 9 systems with one or two Solaris 10 boxes hanging around looking cool.  By the time I left the university five years later, we had migrated nearly our entire catalog of supported applications into zones running on Solaris 10 with ZFS backing the whole enchilada.  During this migration, my cohorts and I rarely practiced the dark arts of dtrace.  Only when things went really wonky did we start writing D to figure out why.

Herein lies the mark I think we all missed while discussing Jarod’s points yesterday.  First and foremost, yes, dtrace is difficult to learn but so is system administration and that hasn’t stopped a lot of smart people from doing it everyday.  Clearly, being difficult is not an insurmountable barrier.  There is a second issue which I believe is the real deal breaker for dtrace adoption; the fact that dtrace isn’t truly needed that often.  I mean, “hard to learn” and “don’t need it that often” is a hard sell to any resource constrained system administrator.  My team at the university and I fell back on dtrace only when all other tools failed to do the job.  In our view, the vast majority of system administration problems can be sovled with well worn tools like iostat, prstat, mpstat, snoop, vmstat, mdb, etc.

I believe it was Bryan who mentioned that you become a dtrace convert once it pulls your ass out of a fire once or twice.  That was certainly the case for myself.  I solved some rather wonky problems and got to be the hero a couple of times.  Thusly, I fell in love with all the power dtrace affords and became a convert.  Behind the scenes, however, I had to work damn hard to earn a pulls asses out of fires with dtrace achievement.  The majority of the time I spent earning the achievement was at home and not during work hours.  Things like like LDAP clusters needing attending and new Oracle DB instance builds constantly get in the way of learning.

My dtrace abilities are founded on spending a lot of time with my nose stuck in Richard McDougall and Jim Mauro’s Solaris Internals book.  Serious dtracing requires serious understanding of the OS you are instrumenting.  I don’t think many professional administrators in companies that are just trying to keep the lights on have the time nor the inclination to climb the wall of dtrace since they can solve the vast majority of their system problems with well worn tools (or hire Jarod).  Quite frankly, I don’t believe I’ve ever met another administrator that was interested in learning dtrace that didn’t first have an obsessive devotion to computing that started at a young age.  This describes me and probably most of the other folks at dtrace.conf yesterday.

(An entertaining thought entered my head shortly after I wrote this post.  If the Oracle guys get their Linux dtrace port working well, we may see a significant uptick in dtrace adoption due to the fact that the well worn Linux stat tools are not as complete as their Solaris counterparts.)

I’d like to hear what other folks in the community think.  I’ll open up the comments and make an attempt to keep the spammers down.

Posted in computers, illumos, solaris | 6 Comments

My Illumos Development Workflow

Lately, I’ve been thinking about how to setup a development workspace to tinker with Illumos builds as they come rolling hot off github.  Ideally, I’d prefer to code on my laptop (macbook), build on my server (orion), and test on a virtual system (oi-dev-box).  Luckily, being mostly finished with our big move westward has afforded me some time to sit down and play with Illumos again.

I’ve been using OpenIndiana KVM for a little while now as my primary hyperviser at home and I’ve been very pleased with its ease of use and stability. (You know you’re a nerd if you ever use the phrase “primary hyperviser at home” seriously.)  I used to run systems in VirtualBox but the overhead associated with VirtualBox VMs was a little annoying.  KVM + ZFS is quick and I don’t have to bother with a lot of the VirtualBox-isms to manipulate and test simple VMs.

Yesterday, I browsed over to the Illumos build document to see if much had changed since the last time I logged in.  I have to say, “Wow!”  What a difference a few months makes.  The build document now contains an excellent write up on how to build Illumos on one system and test it on a different system.  This is a much easier method to test with than building on a local system, applying the update with onu, and rebooting.  Especially if you’re like me and your build server is the fastest machine in the house and you’ve got it doing more things than just building Illumos.

I setup my basic development environment like this:

  1. [Orion] Clone Illumos source tree from github with git.
  2. [Orion] Create an experimental branch on the build machine’s Illumos git repository.
  3. [Macbook] Clone the experimental branch from the build server to the laptop.
  4. [Macbook] Edit Illumos source.
  5. [Macbook] Push the Illumos source back into the build system.
  6. [Orion] Build the source.
  7. [Orion] Make sure the package depot for your build is functional.
  8. [OI-dev-box] Update with your latest changes.

I know that seems like a lot to create a build environment but much of it is only setup that just needs to be done once.  The actual workflow goes much faster once you’re familiar with the tools and process.  So, lets tackle each one of these points separately.

1. [Orion] Clone Illumos source tree from github with git.

This one is easy to take care of.  Simply download the Illumos source to a system you’re going to build with.  The Illumos build document has more information on how to prep and configure a machine to build Illumos.

orion$ git clone git://github.com/illumos/illumos-gate.git

2. [Orion] Create an experimental branch on the build machine’s Illumos git repository.

With git, branching and merging is dead simple and your coding life much easier.  It’s also easy to make patches of code you’ve just modified if you work out of your own branch. In fact, I generally create branches most of the time I am writing new code so I can either merge the branch or just delete it.  The first thing I do after cloning Illumos is create my own branch to tinker with:

orion$ git checkout -b experimental

You should now have two branches. A master and experimental. You can double check which one you’re currently using by executing:

orion$ git branch
* experimental

3. [Macbook] Clone the experimental branch from the build server to the laptop.

Now that I’ve created an experimental branch on my build system, I need to clone the experimental branch to the system I plan to edit the source on.  Use an SSH key between the two systems since you’ll likely be committing changes to your build system often.  The following command will pull the experimental branch to your local system:

macbook$ git clone -b experimental ssh://orion/code/illumos-gate

4. [Macbook] Edit Illumos source.

I can now modify the illumos source code on my laptop with an editor of my choosing (macvim) and push the changes into my build system.  Here’s a trivial example that’s easy to test:

macbook$ cd illumos-gate
macbook$ vim usr/src/cmd/find/find.c

Add some silliness to find.c that’s easy to test. Something at the beginning of main() like:

fprintf(stderr, "I found find!\n");

5. [Macbook] Push the Illumos source back into the build system.

Now you need to add your modification to your repositories’ staged changes, check the status of the change, commit the change, and push it into your build machine.

macbook$ git add -u
macbook$ git status
macbook$ git commit -m 'a simple change to find'
macbook$ git push

The change has now been pushed into your build system on the experimental branch. To verify, log into your build machine and make sure you’re in the experimental branch.

orion$ cd illumos-gate
orion$ git checkout experimental
orion$ vim usr/src/cmd/find/find.c

Hopefully, your modification made it over to the build server. Now it’s time to build Illumos and test the change on a VM.

6. [Orion] Build the source.

I’m not going to go over setting up a build system for Illumos since the process is well documented in the Illumos build document.  You’ll want to make sure you’re on the experimental branch of the build system’s Illumos repository before starting, however.

orion$ cd illumos-gate
orion$ git checkout experimental
orion$ ./nightly.sh -i illumos.sh

7. [Orion] Make sure the package depot for your build is functional.

The Illumos build process packages the files you just compiled and linked into IPS images.  These images are then copied into a file repository on your build system.  If you share this repository across a network, any system using Illumos and IPS can update from it.

The easiest way to setup a network repository is to start the pkg.depotd listener manually and point it at your new Illumos file repository.  (If you’re going to do this a lot, you’ll want to setup the repo server in SMF as documented in the Illumos build document.)

orion$ sudo /usr/lib/pkg.depotd \ 
-d /code/illumos-gate/packages/i386/nightly/repo.redist/ -p 8151 \
--proxy-base http://"`hostname`.`domainname`":8151/on-nightly

In this example, I started the depotd listener on port 8151 and named the repository on-nightly.

8. [OI-dev-box] Update with your latest changes.

Next log into your test VM and create a new boot environment:

oi-dev-box$ sudo beadm create nightly

Now mount the boot environment to /a in the VM:

oi-dev-box$ sudo beadm mount nightly /a

Set your new repository as the primary publisher for your new boot environment:

oi-dev-box$ sudo pkg -R /a set-publisher -P -e --no-refresh \
-O http://orion:8151/on-nightly on-nightly

Downgrade the current publisher for the new boot environment:

oi-dev-box$ sudo pkg -R /a set-publisher --no-refresh \
--non-sticky openindiana.org

Refresh the packages in your boot environment:

oi-dev-box$ sudo pkg -R /a refresh --full

Uninstall the entire image so that you’re able to update piece parts of the Illumos system:

oi-dev-box$ sudo pkg -R /a uninstall entire

Now, update Illumos on the new boot environment!

oi-dev-box$ sudo pkg -R /a image-update

Once the new boot environment is done updating, activate it with:

oi-dev-box$ sudo beadm activate nightly

Reboot the machine. Hopefully, it boots! Now attempt to run your modified find command and see the results:

I found find!

As long as you’re using this boot environment in your VM, you don’t have to follow all the steps above every time you want to make a change to your Illumos source. You only need to push the changes into your build machine, rebuild, and run pkg image-update on the VM.

This blog post will probably be updated in the future as I learn new ways to improve my development process.  Let me know if you have any trouble or questions with this process by either leaving comments or tweets at @Graftolistic.

Have fun with Illumos!

Posted in computers, development, illumos, openindiana | 1 Comment

Settled in Silicon Valley

deer tick (ixodes scapularis)

It’s been several months since I’ve posted to this blog. In that time, Sara, Alex and I picked our stuff up and moved from Texas to California. This is the second time my wife and I have moved half way across the country in the past 10 years (first time around it was from Ohio to Texas for Sara’s job with the University of Texas at San Antonio). This time, I took a job with Apple, Inc. in Cupertino.

This past weekend, we finally unpacked the plethora of boxes that filled our apartment. I have to say, I was a bit shocked by how much rent averages around here. In Texas, an apartment of the same square footage goes for less than half of what we’re paying here. Then again, so do most of the salaries.

To take a break from the marathon unpacking session my wife and I were attempting this past Saturday, I decided to take the family into the Santa Cruz mountains for a picnic and a small hike. (Alex calls mountains ‘mounts’ which is almost as cute as him calling buttons ‘bunts’ as in, “Daddy, I want to wear a bunt shirt today!”.  Three year olds are hands down the funniest and most frustrating people on the planet.) The hike was awesome! Shortly thereafter, however, I had a run in with my first California bug bite.

I was taking more boxes to the trash when I felt a strange sensation on the side of my ribcage under my right arm. I went inside and looked in the mirror and found a deer tick!  Luckily, I’ve been bitten by ticks in the past and knew not to grab it and pull it out quickly.  Doing so will generally leave a piece of mandible in your skin.  Ticks can carry nasty bacterium like the spirochete that causes Lyme disease and it’s probably best not to leave infected bug parts inside of you.  Instead, grab the little bugger with tweezers as close to the jaw as possible and gently pull it out.

Once Sara got the tick out of my side, I put it in a baggie and took it to a crusty old doctor who said “Yup, that’s a deer tick.  Here’s some Doxycycline.”  Antibiotics FTW!

So that’s my first bug experience in California.  I’m looking forward to more of the computer kind rather than kind with legs.  Since this is an Illumos blog, I will mention that I finally unpacked my Illumos build machine and promptly compiled the latest repo update.  This time with GCC!  The result?  Hard reset during kernel loading, probably a driver.

Mounts and bunts in one photo!

Posted in california | 1 Comment

Update: Extended Page Table (EPT) Required for Illumos KVM!

The Intel Extended Page Table (EPT) feature is required for Illumos KVM.  That means you’ll need a recent Intel processor (Core i3, i5, i7 or newer Xeon) to use KVM on OpenIndiana or SmartOS.  Unfortunately, a processor that supports VT extensions alone is not enough.  Folks with Core 2 based systems will not be able to use Illumos KVM.

Joyent has good information on KVM and its requirements on their SmartOS github: https://github.com/joyent/illumos-kvm

Generic information on EPT: http://en.wikipedia.org/wiki/Extended_Page_Table

Posted in Uncategorized | Comments Off

Installing KVM and Creating a Debian VM in OpenIndiana 151a

KVM (Kernel-based Virtual Machine) was developed in Linux as a very speedy in-kernel virtualization solution that takes advantage of Intel’s VT and AMD’s V technology. Recently, it was ported to Open Solaris derived operating systems such as Joyent’s SmartOS and Open Indiana.  Currently, libvirt is not supported so creating a virtual machine is a bit of a manual process.  In this post, I will describe my process for creating KVM based virtual machines on my Open Indiana 151a server.  I will describe how to install a Debian Linux system.

First download an install ISO of the operating system you’d like to install.  I am fortunate enough to work for a university with access to Internet2 and I like to download ISO’s from the University of Texas.  Here is their Debian mirror.

Now, on your Open Indiana 151a server, install the KVM module, application, and driver packages by executing:

# pkg install driver/i86pc/kvm \
system/qemu \

Next, if you’re using ZFS, create a zvol for your virtual machine hard drive. On my server, I have a mirrored zpool named local that I plan to install my virtual machines on.

# zpool status
  pool: local
 state: ONLINE
  scan: none requested

        NAME        STATE     READ WRITE CKSUM
        local       ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            c2d1    ONLINE       0     0     0
            c3d1    ONLINE       0     0     0
            c5d0    ONLINE       0     0     0

errors: No known data errors

  pool: rpool
 state: ONLINE
  scan: none requested

        NAME        STATE     READ WRITE CKSUM
        rpool       ONLINE       0     0     0
          c4d0s0    ONLINE       0     0     0

errors: No known data errors

local is one vdev of three mirrored 1TB drives as recommended for home servers in this great blog post by Constantin Gonzales. I want to create my virtual hard drives on this zpool.

# zfs list -r local
NAME                                     USED  AVAIL  REFER  MOUNTPOINT
local                                    256G   657G    31K  /local
local/shared                             256G   657G   172G  /shared
local/shared/Virtual                    83.8G   657G    33K  /shared/Virtual
local/shared/Virtual/ISO                3.56G   657G  3.56G  /shared/Virtual/ISO
local/shared/Virtual/KVM                80.2G   657G    43K  /shared/Virtual/KVM

I decided to create a 10 gigabyte volume for my Debian virtual machine hard disk to live on.  Here’s the command I used to create it:

# zfs create -p -V 10G local/shared/Virtual/KVM/debian/disk0

The -p option creates parent directories (if they don’t already exist) underneath the given ZFS dataset path. -V tells ZFS this dataset is a zvol and 10G says it’s going to be 10 gigabytes in size. You can see the new zvol by executing the zfs list again:

# zfs list -r local
NAME                                     USED  AVAIL  REFER  MOUNTPOINT
local                                    292G   622G    31K  /local
local/shared                             292G   622G   174G  /shared
local/shared/Virtual                     117G   622G    33K  /shared/Virtual
local/shared/Virtual/ISO                26.7G   622G  26.7G  /shared/Virtual/ISO
local/shared/Virtual/KVM                90.6G   622G    46K  /shared/Virtual/KVM
local/shared/Virtual/KVM/debian         10.3G   622G    31K  /shared/Virtual/KVM/debian
local/shared/Virtual/KVM/debian/disk0   10.3G   632G    16K  -

Now, we need to create a virtual network interface for the virtual machine to use.  Use dladm to list your current virtual network devices:

# dladm show-vnic
LINK         OVER         SPEED  MACADDRESS        MACADDRTYPE         VID
vnic0        igb0         1000   0:0:dc:79:f6:e3   fixed               0
vnic1        igb0         1000   0:0:dc:79:f6:e4   fixed               0
vnic2        igb0         1000   0:0:dc:79:f6:b8   fixed               0

If you do not have any virtual network devices, nothing will be listed. Go ahead and create a new one for your new Debian instance using the physical network interface the virtual machine will be communicating on (mine is igb0, your’s could be e1000g0, etc…):

# sudo dladm create-vnic -l igb0 vnic3

List the virtual network interfaces once again and make sure your new device is listed:

# dladm show-vnic
LINK         OVER         SPEED  MACADDRESS        MACADDRTYPE         VID
vnic0        igb0         1000   0:0:dc:79:f6:e3   fixed               0
vnic1        igb0         1000   0:0:dc:79:f6:e4   fixed               0
vnic2        igb0         1000   0:0:dc:79:f6:b8   fixed               0
vnic3        igb0         1000   2:8:20:28:d8:67   random              0

Now create a start-up script for your new virtual machine. Below is the simple script I use to start my virtual machines. For this new Debian instance, I set the CD and HD variables to the Debian install ISO and my new zvol respectively. The VNIC variable is set to the new virtual network interface we created above. I set the memory to 1024MB and the VNC session number to 5.



MAC=`dladm show-vnic -po macaddress $VNIC`

/usr/bin/qemu-kvm \
-boot cd \
-enable-kvm \
-vnc$VNC \
-smp 2 \
-m $MEM \
-no-hpet \
-localtime \
-drive file=$HD,if=ide,index=0 \
-drive file=$CD,media=cdrom,if=ide,index=2  \
-net nic,vlan=0,name=net0,model=e1000,macaddr=$MAC \
-net vnic,vlan=0,name=net0,ifname=$VNIC,macaddr=$MAC \
-vga std

The /usr/bin/qemu-kvm executable requires root level permissions so I run the script with sudo. Like this:

# sudo ./start-debian.sh

If all goes well, the virtual machine should boot. A windowed screen with a graphical interface will not appear like in VirtualBox or VMWare. You’ll only see a bunch of text like this scroll by:

drive 0x000fda80: PCHS=16383/16/63 translation=lba LCHS=1024/255/63 s=20971520
Running option rom at cb00:0003
ebda moved from 9fc00 to 9f400
Returned 53248 bytes of ZoneHigh
e820 map has 7 items:
  0: 0000000000000000 - 000000000009f400 = 1
  1: 000000000009f400 - 00000000000a0000 = 2
  2: 00000000000f0000 - 0000000000100000 = 2
  3: 0000000000100000 - 000000003fffd000 = 1
  4: 000000003fffd000 - 0000000040000000 = 2
  5: 00000000feffc000 - 00000000ff000000 = 2
  6: 00000000fffc0000 - 0000000100000000 = 2
enter handle_19:
Booting from Hard Disk...
Boot failed: not a bootable disk

enter handle_18:
Booting from DVD/CD...
150MB medium detected
Booting from 0000:7c00

From your workstation, use a VNC client (I use TightVNC) and connect to your KVM server using the session number you setup above. On my Linux workstation, I run:

jgrafton@pod:~$ vncviewer orion:5

If all has gone well and your KVM server firewall is not blocking port 5905, (5900 + your VNC session number) the VNC client should connect to your KVM server and you should see something similar to this screenshot:

Install Debian and enjoy your new KVM virtual machine!  Remember, you’ll need a separate VNIC and VNC session number for each new virtual machine you create.  Have fun!

Let me know if it works for you by either leaving comments or tweeting me at @Graftolistic.

Posted in computers, openindiana, solaris | 7 Comments