Rock the Fat Knot

I’m a hard core vi user. You just can’t beat it for a quick file edit. (This is where my friend, Jack Lunn, would argue that Emacs is better… but I think we can agree that both are faster to crank up than most IDEs.) Recently, I’ve switched to Netbeans for Java development at work. I had been using IntelliJ and it almost drove me insane. (See my post about the Corrupted Cache.) Netbeans has been working well for me, though. (It’s no vi, but it does have features that are very useful when working with large projects.) With Netbeans 6.5, you can get Python support. I’ve been using this for a few weeks now. It’s nice to have all of my Java/Python code open in one IDE. This morning, I ran across a screencast showing some upcoming code coverage support for Python in Netbeans. This looks impressive.

I’ve been using OpenSolaris 2008.05 at work since it was made available. I use it as my primary OS on my work laptop (an HP Compaq 6910p My wife refers to this laptop as “the ugliest laptop in the world.” Compared with the Apple products she’s used to working on, the HP is not very attractive.) From the beginning, I’ve had issues installing/updating packages with the new IPS packaging system. I’m not dissing IPS. I just ran into a lot of issues trying to run pkg install and pkg image-update. Other people have reported similar issues with timeouts and such. Anyway, in a recent attempt to update my version of pkg, I managed to hose up my system. Having so much trouble at the command line with pkg, I decided to try out the Package Manager GUI. It allowed me to just select the latest version of pkg and install it. Everything processed successfully. I was excited! I closed the Package Manager GUI and went to the command line to try out my new and improved version of pkg. It didn’t work. Even worse, I started having issues with ls and every other command I tried. Core dumps were running wild on “the ugliest laptop in the world.” From another machine, I went to google and started searching to see if other people had encountered similar issues. I found one post that lead me to believe I had “updated too much.” I couldn’t find any posts that described how to recover from this. Determined not to give up on OpenSolaris, I started trying to get my laptop back in order. I didn’t want to lose my home directory. I have tons of valuable code snippets, settings, documents, and huge subversion checkouts of various projects and branches of those projects (about 80GB of data total). So, I’m including the steps I followed to get my laptop running again (along with all of the data in my home folder) below. There may be a better way to do this, but this worked for me.

Step #1: Boot from an OpenSolaris Live CD Boot up with an OpenSolaris CD. We’re going to use the live CD to help us backup the home directory. Step #2: Backup your home folder Open up a terminal and execute the command:

pfexec zpool import -f -R /tmp/rpool rpool

Then take a snapshot of your important data:

pfexec zfs snapshot rpool/export/home@pre-reinstall

I didn’t have an external drive with enough free storage space, so I shipped the backup over to another machine via ssh:

pfexec zfs send rpool/export/home@pre-reinstall | ssh user@myhost.com " \
          cat > ~/homedir.snapshot"

Step #3: Use the OpenSolaris Live CD to do a Full install of OpenSolaris With my home directory safely backed up, I wiped the machine and did a reinstall of OpenSolaris. Step #4: Boot into single user mode (login as root)

ssh user@myhost.com "cat ~/homedir.snapshot" | \
pfexec zfs recv -F rpool/export/home

Your home directory should be ready to go now. Reboot and enjoy.

Update: I recently used pfexec pkg image-update to update my 2008.05 installation to 2008.11. OpenSolaris 2008.11 is awesome! I’m loving the Time Slider functionality. The pkg image-update worked flawlessly. My upgrade was a complete success. None of my tools or applications were broken during the upgrade. The folks working on OpenSolaris have really been putting in some hard work to resolve a lot of the early issues that I experienced.

I’ve been reading Simon Willison’s blog for quite a while now. He frequently posts interesting ideas and links. Yesterday, I noticed that he had a link to videos from DjangoCon 2008 on YouTube. So far, I’ve watched Schema Evolution and Reusable Apps. I learned a lot from both videos. I’ve only been working with Django for a few months (mostly for this blog) and I’m really enjoying it. I think it’s really cool that they’ve posted videos from the Django conference so people that couldn’t attend (the conference or a particular talk) can check them out (and learn something). Living in rural South Carolina, there are not (currently) a lot of local opportunities to exchange ideas with other python developers. I hope that more (python related) conferences follow Django’s lead and post conference videos. Great job!

Recently, the hard drive in my trusty Mac Mini decided it was going to have problems. Running some diagnostic tests revealed that it has some bad sectors. I had another firewire drive on the shelf, so I plugged it in and booted from my Leopard DVD. The Leopard Installer allows you to access several utilities including the Time Machine recovery functionality. So, I selected my Time Machine drive and asked the utility to recover my machine to my spare firewire drive. In a little under two hours, my machine was up and running again — booting from that spare firewire drive. The recovery process was simple and worked flawlessly. I’ve never been very good about properly backing up my data. Thanks to Time Machine, I don’t have to worry about it any more.

Rock the Fat Knot was recently migrated from WordPress to Byteflow. This migration also included moving the site to a Joyent Accelerator running OpenSolaris. I wanted a good way to develop and test things out locally, so I created an OpenSolaris virtual machine on my Mac using VirtualBox. I used an OpenSolaris 2008.05 disk for the install. It’s a newer build than what’s running in my Accelerator, but I’m OK with a close approximation for this work. I was more concerned with running the same Python, mod_python, Apache, and MySQL builds. I followed this guide to get OpenSolaris up and running in a virtual machine. I had difficulties getting the VM’s network connectivity working. I tried changing the VM’s network adapter to “Intel PRO/1000 MT Desktop (NAT)” and it worked. (Note: I’m using about a one year old Mac Mini. PC Net may work for others.) To get name resolution working, I had to add the entries from my mac’s /etc/resolv.conf to the /etc/resolv.conf in the VM. I wanted to be able to access my Apache test server from the Mac, so I used VirtualBox’s VBoxManage program to setup some port forwarding. I ended up mapping host ports 2222 for SSH and 4444 for http (no particular reason for choosing those ports). I called my VM “OpenSolaris2008.05”, so I ran these commands from a terminal to configure the port mapping.

VBoxManage setextradata "OpenSolaris2008.05" \
           "VBoxInternal/Devices/e1000/0/LUN#0/Config/httpd/GuestPort" 80
VBoxManage setextradata "OpenSolaris2008.05" \
           "VBoxInternal/Devices/e1000/0/LUN#0/Config/httpd/HostPort" 4444
VBoxManage setextradata "OpenSolaris2008.05" \
           "VBoxInternal/Devices/e1000/0/LUN#0/Config/httpd/Protocol" TCP

VBoxManage setextradata "OpenSolaris2008.05" \
           "VBoxInternal/Devices/e1000/0/LUN#0/Config/ssh/GuestPort" 22
VBoxManage setextradata "OpenSolaris2008.05" \
           "VBoxInternal/Devices/e1000/0/LUN#0/Config/ssh/HostPort" 2222
VBoxManage setextradata "OpenSolaris2008.05" \
           "VBoxInternal/Devices/e1000/0/LUN#0/Config/ssh/Protocol" TCP

These commands map port 4444 on the Mac to port 80 of the OpenSolaris VM. Likewise, port 2222 on the Mac is mapped to port 22 of the VM. Now, I can test the site from Safari and Firefox on the Mac by navigating to http://localhost:4444. I can also ssh to the vm using a command like this:

ssh -oPort=2222 vmuser@localhost 

With a VM up and running, I exported my Wordpress blog data to a WordPress eXtended RSS (WRX) file. Using WXRImporter.py from Greg Heartsfield (see this thread on the byteflow-users list for usage), I imported my blog data in the WXR file into my byteflow database. To ensure that people didn’t lose my RSS feed, I added a few lines to my Apache configuration:

RewriteEngine On
#Redirect Wordpress RSS Feed to Byteflow
RewriteCond %{QUERY_STRING} ^feed=rss2$ [NC]
RewriteRule ^/$ feeds/rss/blog? [R=301,L]

This ensures that people using my old feed URL still find the latest posts. I also added this rewrite rule in my dev VM for sanity (I seemed to be getting strange name resolution problems without it):

RewriteCond %{HTTP_HOST} ^localhost$ [NC]
RewriteRule ^(.*)$ http://127.0.0.1$1 [R=301,L]

The dev VM setup has been working out very well for me. So far, I’ve not run into any issues when deploying changes out to my Accelerator. Testing things out locally, in a VM that’s pretty close to my Accelerator, is very nice. I’ve also been using VirtualBox at work to run a Windows VM. It’s shaping up to be a very nice product. It includes a nice set of command line utilities. Anyone out there got tips for developing/testing apps that are deployed on a Joyent Accelerator? I know Joyent probably wouldn’t mind selling you a development accelerator, too.

My uncle contacted me this week about problems with his PC. He has a Dell running Windows XP. (No, I have not been successful in converting all of my relatives over to non-Windows systems.) He was having problems starting his machine after upgrading his virus scanning program (AVG). I have used various Linux Live CDs in the past to recover data from machines, so my plan was to use one to get his machine going again. I had a Knoppix CD laying around, but it was a little dated, so I decided to try something new. I did a quick google and found Trinity Rescue Kit (TRK). It was a relatively small download (about 100MB) so I grabbed a copy. It booted right up on my uncle’s PC and even offered the option of exposing his drive as a SMB share. I was able to tar and bzip his important files, and then copy them over to my Mac for archiving to DVD. (The Mac allowed me to easily find and browse my uncle’s SMB shares that were set up by TRK) Once I knew his data was safe, I reinstalled XP and added the programs he would normally use. His machine is back in good working order. Not sure what caused his problems originally — but it could be something related to this. TRK also includes virus scanning and partitioning tools if you think you might be battling a virus or some other disk issue. If you ever encounter a similar situation, I highly recommend a live cd like TRK to help get things going again.

Happy 4th of July!

To celebrate Independence Day, rockthefatknot.com moved to a new home and got a new look. The site is now hosted by Joyent on a Joyent Accelerator. A while back, I posted about moving as much of my stuff to Python as possible. Now, rockthefatknot.com is running on byteflow (Byteflow is a blog engine that’s written in Python using Django.). I spent a couple of nights this past week working to migrate all of the posts over to byteflow. I got some help from the byteflow-users mailing list and used a handy script from Greg Heartsfield to migrate the data over. I setup an OpenSolaris VM in VirtualBox on my mac to handle all of the migration work. I’ll probably write up another post later with the technical details on the migration. Since the migration is complete, I’m going to celebrate!

Yet another reason why I prefer dynamic languages…

Several people have asked me what I’ve been doing with all my spare time now that I’ve stepped down from being a manager (I returned to a strictly engineering role)… Well, I’ve been spending more time with my wife (and our two dogs), eating dinner at a reasonable time and going to sleep at semi-reasonable times. I’ve also been spending a lot of time working with OpenSolaris (my new favorite x86 OS). ZFS and DTrace just plain rock. I’ve also been tinkering with Django. If you don’t know, Django is a dominant python framework for cranking out dominant web sites. I’ve been enjoying spending time working with it simply because it’s the exact opposite of some of the J2EE stuff I’ve been working with the past few years. I don’t need 1 or 2 GBs of RAM to edit a Django project nor do I need 4GB of RAM to run/debug it. I’ve finally decided that that all of my personal stuff is going to be done in just python. I’m going to be converting rockthefatknot.com over to Django in the coming months. I’m also going to be switching up from subversion to Mercurial (because it’s cool and also because it’s implemented in Python!)  After all, Python is the official language of The Markovian Powerhouse (Alex Markov).  Don’t worry — I’ll still be posting Fat Knots.  In fact, I’ve got a special treat coming soon:  Fat Knot Flashbacks…

We’ve been battling some wild and crazy error messages at work, but the error message I got while attempting to pay my phone bill online today takes the top spot: Message from the NSAPI plugin: Cannot open TEMP file ‘/tmp/_wl_proxy/_post__19981_24579’ for POST of 2807 bytes Build date/time: Oct 4 2003 18:00:57 Change Number: 291942 Based on my conversation with the support staff, this means “You need to reset your password.”  Sure enough, after changing my password, the error went away and I was able to pay my bill.   Please help me lead a worldwide effort to rid ourselves of bad error messages.  They eat away at our lives :-)