Platform Engineers or Rock Star Engineers

I've hired a lot of engineers over the years, and one of the first things I'm always asked to add to the job description is "must have experience in XXXX platform." I sometimes put it on there to make people happy, but I rarely will disregard an engineer because they do not have a lot of experience in a particular platform. Give me a Rock Star, or even a very good/great engineer, and I guarantee you that person will run circles around the average engineer with platform experience. Now, if you can find the Rock Star with platform experience -- BONUS! The only time I deviate from this plan is if the project calls for someone to "just get it done and fast." Then I need to pay more attention to platform experience because I don't have time to wait for the new platform to be learned. I've lost track of how many times this simple fact has been proven to me. The qualities of a great engineer carry over to any platform, and a great engineer will pick up a new platform quickly -- mostly because they love learning new things. If you're starting out in software development, concentrate on being a great engineer. That's far more valuable than an engineer that knows a platform.

What makes a great engineer? To me, it's pretty simple. You have a passion and ability to craft outstanding, maintainable, and testable code. You know your algorithms, design patterns, and data structures like the back of your hand. Finally, you posses the other skills necessary to round things out -- communication, time management, risk assessment, strategic and detailed design, and quick decision making (and sticking to those decisions). You can write a loop in Java? I don't really care. You understand dependency injection or how an outer join works. Now we're talking.

What do you think?

Bletchley Park Snubbed by British Government

UK Snubs Support for Home of WWII Enigma

Here we go again. Bletchley Park continues to get little love. Here we have what is basically one on of the birthplaces of modern computing. On top of that, the group of people that worked here, along with their US partners in Building 26, did more to shorten WWII and save countless lives than just about any other group. Makes your life as a military leader a whole lot easier when you know what your counterpart is up to.

It seems now that the UK will not give Bletchley Park the same status as Imperial War Museum.

"We have no plans at present to associate it with the Imperial War Museum," Lord Davies said. "The House is all too well aware of the significance of designating any area in association with a museum of that rank, but I want to give an assurance that Bletchley Park will continue to develop under the resources made available to it."

OK, I know I'm a little biased because I'm a history buff, but I'm also aware of the history of my profession. Without these two groups of scientists, we might not have the same level of computing we have today. This is where Alan Turing (of Turing Machine fame) cut his chops.

Let's not forget about these people and what they did. I know it was super-secret, but it was almost 70 years ago now. 

 

Workling Timing Issue

As you probably noticed already, I use Workling a lot, and I wrote about it a few times (Part 1, Part 2, Part 3). One minor gotcha to be aware of is that you need to make sure you handle it when Workling is too fast. A common use of Workling is to make a call in an after_save method. For example:

class User < ActiveRecord::Base
  def after_save
    MyWorker.asynch_geocode_address(:user_id => self.id)
  end
end
class MyWorker < Workling::Base
  def geocode_address(options = {})
    user = User.find(options[:user_id])
    user.geocode_address
  end
end

Pretty straight forward, right? Unfortunately, what you will find is that often your worker will get called so fast on create that ActiveRecord hasn't committed the transaction yet, and User.find will fail. To get around this, you need to write your workers that could be called from after_save/after_create to handle this condition. Instead of the above, you need to do the following:

class MyWorker < Workling::Base
  def geocode_address(options = {})
    retries = 3
    begin
      user = User.find(options[:user_id])
      user.geocode_address
    rescue => err
      if (retries -= 1) > 0
        sleep(0.2)  # Give ActiveRecord a chance to save
        retry
      end
      raise err
    end
  end
end

Now we catch the exception that ActiveRecord throws when it can't find the new user yet. Waiting a short time and trying a few times gives ActiveRecord a chance to commit the record.

Enjoy asynchronous processing...

Gotcha with find_each and find_in_batches

Rails 2.3 added a couple of nice new methods - find_each and find_in_batches. Both methods accomplish the same thing in a slightly different way. Unlike a normal finder these methods grab objects in batches instead of all at once. For instance, if you have 500,000 users, you don't want to do the following:

User.find(:all).each { |user| user.some_method }

The reason is that you just loaded all 500,000 records into memory, and your server is not happy. Instead, you could do:

User.find_each { |user| user.some_method }

By default, the above will only load 1,000 User objects into memory, and your server will thank you. If 1,000 is too big/small for you, use the :batch_size option to change it. The find_in_batches method is similar except that it provides the array to the block instead of one object at a time. For example:

User.find_in_batches do |users|
  users.each { |user| user.some_method }
end

If you ever used the wonderful will_paginate gem, you are probably familiar with the concept from the paginated_each method that will_paginate provided.

So, what is the problem? The problem is that you have to aware that unlike paginate_each, find_each and find_in_batches work by setting up a with_scope block. Therefore, if you need to do any other finds on that same model, the scope will apply. Usually this only affects relationships, but it isn't hard to forget. Here is an example:

# This is a purely made up example
class User < ActiveRecord::Base
  # There is a last_login_at attribute
  named_scope :recent_login,
              lambda { |*args|
              { :conditions => ["people.last_login_at >= ?", (args.first || 1.week.ago)] } }
  belongs_to :parent, :class_name => "User", :foreign_key => "parent_id"
end 
User.recent_login.find_each do |user|
  parent = user.parent # This will include the recent_login scope.
end 

It's no different than other with_scope issues, but it isn't as obvious. You can get around it by doing:

User.recent_login.find_each do |user|
  # Got use send because with_excusive_scope is protected.
  User.send(:with_exclusive_scope)
    parent = user.parent # This will include the recent_login scope
  end
end

Now, go and be kind to your server with find_each and find_in_batches - just remember the scope.

Why We Need Audiophiles

Gizmodo - Why We Need Audiophiles - Audiophiles This is a great read for anyone that is interested in great sound. I've said it a few times, I'm not into expensive audio gear because I like gadgets (although that part is fun too). I'm into expensive gear because I flat-out love music, and I want to hear it like the artist meant it to be heard. I will tolerate MP3's on my phone, but not when I want to hear some really choice music played like it was meant to. Sadly, between my kids and my schedule, I do not get near enough time for pure listening.

This is Michael Fremer. He's listening to "Avalon" by Roxy Music on his $350,000 stereo system. It sounds excellent. He's a bit crazy, but if you love music, you need him.

BTW, "Avalon" is one of those pieces of music that screams for high quality. My system isn't quite in the league of $350k, but it does pretty well.

I think the following quote sums up my thoughts exactly:

Because the thing is, Fremer loves music first and foremost. The audiophile I had feared was one who cares far more about the overpriced gadgetry than the actual music. This is not who I ended up meeting. This man listens to music and makes sure it was recorded with the best fidelity, that the intents of the artist have been preserved. And thank God he does, because we certainly don't.

If you ever get a chance to listen to a favorite, well-recorded piece of music on a choice system, take it. You may be surprised at the subtleties and feeling you are missing. Be careful though. Many recent recordings are mixed for MP3 players and will sound pretty bad on a great system. The really scary part is that some recent "Remastered" versions of classic albums were mastered for MP3 players as well. Many times, it is better to stick with the original.

In a future post, I will describe what I built/coded to control my entire media system from one remote. Note, since I have small children, all my equipment is in a locked closet.

The Perils of Testing Code

Testing code is good, right? No! Testing code is bad. Testing functionality is good. How many of us have written a bunch of code and then turned around and wrote a bunch of tests for that code? Be honest! Everyone has done it, but why is it a bad thing? The problem with this is that far too often you end up testing what the code DOES, not what it SHOULD DO. This has always been one problem with testing your own code. You assume your code is correct, so tests are written that validate the code. Unfortunately, all this does is keep you honest in case you change the code to cause the test to break. You have not proven that the code does what it's supposed to do. Here is a quick example in RoR:

app/models/user.rb:
class User < ActiveRecord::Base
  def unsubscribe!
    add_role(Role::UNSUBSCRIBED)
    forget_me # this will save
  end
end

test/unit/user_test.rb:
class UserTest < ActiveSupport::TestCase
  test "should unsubscribe" do
    user = users(:dave)
    assert user.unsubscribe!
    assert user.unsubscribed? # Assume I have a test for this as well
    assert_nil people(:dave).remember_token
  end
end

Great, we have a test that verifies that the user is unsubscribed. The problem is that when a user is unsubscribed, they SHOULD have had all their email preferences turned off as well. The code is wrong, but the tests will not catch that because the test only verified the code as written.

This is where Test Driven Development (TDD) comes in. With TDD, you write the tests for the functionality you want to happen. Once you described all the desired functionality as tests, you write the code necessary to make all the tests pass. Now, you're testing what the code should do and not what it does. I'm on board with this, but there are a few problems in the real world.

The biggest problem that I have with TDD is that it assumes you know what the code should do in the first place. Unfortunately, when you innovate you don't always know what you will end up with. You're experimenting, trying new things, seeing what sticks. This is normal for many projects. Still, you need to have tests to verify things once you do settle on a direction.

How does this fit into TDD? It doesn't fit directly because you aren't writing tests first. What I do in this situation is the following:

  1. Once I settle on a direction and set of functionality, I attempt to write the tests for what I have without looking at the implementation code. Write the tests to verify your intended results rather than looking at the code to see what it does. I'm trying to avoid the perils of testing the code.
  2. Now that I have a direction, I can go back to writing the test followed by making the test pass.
  3. If a problem is discovered anywhere, I write a test to reproduce the issue before fixing it in the code.

The other issue I have with TDD is how do I write code to verify what code should do if I don't know how it is supposed to do it? When writing the tests, I have to leave many method calls blank until I figure out how things are done and what I should check. Often, I go back to the above until I have a set of standard methods to use, or better yet I let the tests determine what the methods should be. Either way, the above scheme still works for me.

This hybrid approach works for me, but YMMV. The bottom line, as always, is that the code isn't done until the tests are done as well.

Now, go write some tests!

Replacing Mongrel with Passenger (mod_rails)

I've been using Apache + Mongrel for a while now. I read about Passenger when it first came out, but I never felt the need to bother with it since I have a bunch of recipes for Apache and Mongrel (see a previous posts about background tasks with Workling). The other day, I figured I would give it a shot. Now, I think I found my new way to deploy Rails apps. It is so simple, and it works well. Some people write off Passenger because it uses Apache and not Nginx. Nginx is a little faster, and it is the latest "best practice," but I haven't had issues with Apache in forever. As for the speed issues, I don't see it being a big factor to the end users. A few milliseconds is not going to be noticeable, at least for my apps.

Installation could not be simpler.

  1. sudo gem install passenger
  2. sudo passenger-install-apache2-module
  3. Follow the instructions the previous script displays
  4. Restart Apache
  5. Done!

If your development machine is OSX, don't bother to manually change the Apache conf for your application and your hosts file just yet. You can use the Passenger Preference Pane instead. This little beauty will make all the changes to your Apache config files for you. One of the benefits is that you can have dozens of small Rails projects configured, and all are available with their own url -- no need to run script/server for each. For install, the site explains it pretty well, so I won't go into it here.

I did run into one problem on my Leopard machine. My default Apache config file had the httpd process running as the _www user, but my project files are owned by me. The result is that I kept getting permission errors accessing files in my project. I'm sure there is a solution to the problem by changing the permissions on my project files, but I didn't find it. I ended up running httpd as me, and that fixed it. I did not have this issue on my test or production systems.

Even with the problem, the entire process took me less than an hour. What I gained is the following:

  • I no longer have to run script/server every time I switch to a new application for development.
  • All of my apps on my dev machine have their own url (myapp.local, etc.) instead of localhost:3000.
  • My god.rb config no longer has to watch Mongrel processes. It's all in Apache.
  • No Mongrel cluster or load balancing in Apache configs among multiple Mongrels. Apache just operates normally, and my config files are a lot simpler.
  • Mongrel hasn't had any updates in while, but Passenger is still under active development.
  • An added benefit is that my test/staging server performs better since only the applications being used are running. By default Passenger will drop any process not accessed in 300 seconds (you can change that).

All of the required changes involved removing any references to Mongrel in my god.rb config, apache config, and my deploy.rb. The only thing I needed to actually change is how to start/stop/restart my application. Remember, there is no mongrel process to start or stop. It's all Apache. Passenger makes it easy, however. All you need to do is touch tmp/restart.txt. If that file is modified, then Passenger will restart your application.

My updated config/deploy.rb:

namespace :deploy do

  desc "Start the application"
  task :start, :roles => :app do
    run "touch #{current_release}/tmp/restart.txt"
    sudo "god start listeners"
  end

  desc "Stop the application"
  task :stop, :roles => :app do
    # Do nothing for application
    sudo "god stop listeners"
  end

  desc "Restart Application"
  task :restart, :roles => :app do
    run "touch #{current_release}/tmp/restart.txt"
    sudo "god restart listeners"
  end
end

Give Passenger a try. I think you will like it.

Music Is Dead: iPods and Young People Have Utterly Destroyed Music

Music Is Dead: iPods and Young People Have Utterly Destroyed Music This is depressing for someone like me. A Stanford professor did an informal study over several years, and he found that students each year preferred low-quality MP3 more and more. Like the author, this is disappointing for me because it most likely means it will become harder to find high-quality music. It's already hard, but it will get worse. Even CDs are being mixed with MP3 players in mind.

I guess it comes down to what you're used to. If it's all you ever heard, then you don't know what you're missing. Personally, I am spoiled by hundreds of high quality live performances, and I always wanted to recreate that sound at home as much as possible. Low quality MP3s just don't do it for me. They are fine on my iPhone or small player, but when I sit in front of my home system I want to hear like I was in the studio.

I had a similar experience many years ago when I had a top-end VCR with one of the best TVs available at the time. I thought the picture was amazing. Then I hooked up a cheap DVD player, and I forgot all about the VCR. The problem here is that these low quality recordings are becoming so much the norm that it will be hard for anyone to know what they are missing.

What is a music lover to do?

Reflecting on a New Generation of Mirrors

Gallery - Reflecting on a new generation of mirrors - Image 1 - New Scientist Continuing on the theme of interesting optics, Andrew Hicks, a mathematician, has created mirrors that accomplish some amazing feats. Imagine looking at text in a mirror, and the text is not backwards. Imagine 360-degree panoramas without distortion. It's all pretty cool stuff.

All of the surfaces were designed with computers to reflect the light rays across the mirror surface before returning them to the viewer. With the right shapes, you can do some bizarre tricks with light. It's like Fun House mirrors without all the distortion. Some of the applications of this are in robotics and even automobiles.

I remember back when I was getting my masters, I built a 3D rendering engine for giggles (I wanted to understand the math). If you have ever built one of these you know it's all about calculating where light rays go and what color and brightness they end up with when they get there. In this case, you need to work backwards. Here is what I want. Now, what is the shape necessary to get there, and that's a whole different ball game. I wonder if the idea is to solve the same rendering equations in the other direction.

iPhone Wishlist

I've been an iPhone user since the first weekend the 3G came out, and so far I love it. Prior to the iPhone, I was a Blackberry user for a while. My reasons for waiting to change were:

  1. My contract with Verizon conveniently ended the same month the 3G came out.
  2. AT&T had lousy coverage at my home and office. However, for some reason AT&T and Verizon swapped in the months leading up to the 3G coming out. Verizon started to have lousy coverage in my home, and I changed job locations. The new location had lousy Verizon coverage.
  3. The price drop made it a great time to upgrade my wife as well. There was no way I was going to get away with a 1st gen iPhone without my wife getting one as well. With penalties, that would have been $1550 instead of $400. I like Apple stuff, but I'm not that big a fan boy.

Fast forward several months, and I'm not looking back to my Blackberry (ever), but there are a few minor issues with the iPhone that I wish Apple would fix. It's been long enough now that I'm starting to wonder when/if these may happen.

No Indicators on the Locked Page

Why make me slide the bar and enter my code just to see whether I have voicemail or email? It doesn't seem to be that hard to have a couple of indicators on the locked page to let me know. This would be a great time saver and a good compromise away from that annoying blinking LED on the Blackberries.

No Sound Presets

To change any of the sound settings, I have to go in and update each separately. Having something like Night, Office, Silent, etc. would be great. The silent switch is good, but sometimes I really want silence, and others I want the phone to still vibrate.

Related to this, I want different sound settings for each email account as well. For example, I have error emails from my site(s) sent to me. I want those to complain loudly regardless of what I have setup for other volumes.

The Blackberry does this well.

Signatures for Each Email Account

I have three email accounts configured on my iPhone. I prefer to separate my email signatures between work and personal. Again, I could do this on a Blackberry.

One Global Inbox

It would be a lot easier to read my email if there was a single Inbox where all three of my accounts show up. This is what Apple Mail does on my desktop, and I use it all the time. Right now, I have to go through the follow set of interactions for each account:

  1. Tap Personal account
  2. Tap Inbox
  3. Read/Process inbox
  4. Tap Personal to go back
  5. Tap Accounts to go back
  6. Tap Work
  7. Tap Inbox
  8. Read/Process inbox
  9. Tap Work to go back
  10. Tap Accounts to go back
  11. etc.

Lot of tapping going on here! Give me a consolidated inbox, and life would be a lot easier for those of us with multiple email accounts. Did I mention I could do this on my Blackberry?

Notice I did not say copy/paste. In seven months, I think I could have used it twice. I can see Apple making the correct decision to cater to the 80% of users out there and not the reviewers.

Don't get me wrong, I will not go back to a Blackberry without a fight, but give me the above, and I will be an even happier camper.

Excuse me, I have to take a call now...