Ruby Default Hash Value Trick

I came across a neat trick today, when Ruby’s hash is being initialized, it can take a block for default values. This block gets called when a key is requested that’s not present in the hash. This block gets yielded the hash itself and the key that was missing.

This trick is pretty neat because it gives the Hash an oppurtunity to calculate values as needed and to memoize them for future look ups. Consider this factorization example:

factorization = Hash.new do |hash, n|
  upper_bound = (Math.sqrt(n)).floor
  factors = []
  (2..upper_bound).each do |trial_factor|
    if n % trial_factor == 0
      factors << trial_factor
      factors << n/trial_factor
    end
  end
  hash[n] = factors.sort
end

This example calculate all of the factors for the given key and then stores it to the Hash. The second time a factor is needed, it will be looked up from the Hash instead of being calculated. This is handy because calculating factors like this gets expensive fast for large numbers. But consider if you wanted to calculate the prime factors using the above implementation. That might look something like:

    prime_factorization = Hash.new do |hash, n|
         hash[n] = factorization[n].select do |factor|
            factorization[factor].empty?
        end
    end

This (naive) code uses the above factorization data set to calculate prime factorization. Because the factors are stored in the hash it’s really inexpensive to calculate them (like a constant time O(1) inexpensive).

Summer Reading List 2013

Books I read in Summer 2013

It’s November now and summer has definitely drawn to a close, I read some great books this summer. Here’s what I read:

Where Wizards Stay Up Late

Where Wizards Stay Up Late is a history of the founding of the internet. It chronicles the people, places and projects that lead to the internet as we know it today.

As I was reading this I was also spending some time in TCP/IP Illustrated and the RFC for HTTP (and 2.0!) and it gave me a deep appreciation for how much thoughtful work has been done all along the stack.

With so much constant excitement about the future and what’s next I appreciated looking back at how things came to be. Where Wizards Stay Up Late gave me an appreciation Licklider, one of the earliest visionaries of what the web could be.

Watchmen

I have friends that LOVE Watchmen, and having never read through a graphic novel they encouraged me to start here. It was not at all what I thought it was going to be.

Watchmen is a complicated story about living in times of constant war and stress. It’s told in a universe where there are superheros but nothing is fantastical.

I expected that this would be a bright, quick read but it was not. I’m glad I didn’t come into reading it with any expectations, it’s worth letting the story tell itself at its own pace and not have any idea where it’s going to go.

I was fortunate to have not seen the movie or even the trailer until after I completed the book. The movie a disappointment after the deepness of the book.

Zen Mind, Beginners Mind

Zen Mind, Beginners Mind is book about the way our mind works and about living a good life. It’s very personal and I highly recommend it.

Masters of Doom

Masters of Doom is the story of John Carmack and John Romero and the founding of id software. I loved it because you can see the intelligence of both people and they were able to bring together their talents to make amazing things. One of my favorite quotes, from towards the end of the book:

In the information age, the barriers [to entry into programming] just aren't there. The barriers are self imposed. If you want to set off and go develop some grand new thing, you don't need millions of dollars of capitalization. You need enough pizza and Diet Coke to stick in your refrigerator, a cheap PC to work on, and the dedication to go through with it. We slept on floors. We waded across rivers.

It’s also a story of how even two really smart people with common interests can drift apart and end up seeing the world differently.

Shortly after finishing the book, I watched John Carmack’s keynote from Quake Con 2013 and was again impressed with his clarity of intelligence. Carmack is especially interesting to me right because of his involvement with Oculus VR

Coders At Work

Coders at Work some of the best programmers of our time. The book is a series of interviews with the author, Peter Seibel, asking each person many of the same questions. I found this book to be highly quotable and I think I’ve related stories from each interview to coworkers or friends. I especially enjoyed the interview with Jamie Zawinski (jwz). He thinks like I do:

"I find that getting something on the screen as soon as possible really helps focus the problem for me. It helps me decide what to work on next. Because if you're just looking at that big to-do list it's like, eh, I don't know which one I should do—does it matter which one I do? But if there's something you can actually look at, even if it's just the debug output of your mailbox parser, it's like, OK, there!"

JS Everywhere- My First Conference Talk

Nick Rowe Speaking at JS Everywhere

On Friday I gave my first conference talk at JS.everywhere. The talk was about code and fashion and challenged the audience to think of fashion as more than just the clothes we wear by discussing the development journey my team took while we built ModCloth’s Style Gallery. I’ve been working on this talk for about a month and was a little bit nervous because it was different than a normal tech talk. Rather than show off a new piece of technology or demonstrate how to use a framework, I wanted to engage with the audience about an opinion: code is fashion.

Here’s what I learned from getting ready for this talk:

Start early on your talk and get feedback

Conference Badge

I started working on this 15-minute talk about a month ago and that felt like the right amount of time. The talk went through several incarnations as I got feedback from a number of different people. Fairly early on I knew that I wanted to argue that code is fashion but in early talks, someone pointed out to me that I was really saying that fashion is nothing but fads. That was great feedback and because I still had quite a bit of time I was able to think about it and come to a better understanding of what fashion means to me.

Don’t worry so much about your slides

I think that you should be able to deliver your talk even if the slides completely fail and it’s just you up there standing and talking. As an audience member I appreciate talks that use slides to support a point and not to solely make that point. I found that it was much more natural for me to practice saying what I wanted to say and then create slides that supported that. I had a great moment where, after I’d given my talk, someone said “You know you made a great point, it’d be even more powerful if you showed me a picture of that”. Bingo! The talk was in a good place and I was able to add a slide that supported what I wanted to say.

Conference Talk Notes

I chose to use paper notes for this presentation. I’m glad that I did because I was able to think about my talk and make notes no matter where I was. I have a little section of my notes above. It’s all phrases, little pictures, and points I want to remember to make. The different boxes map loosely to slides but for example I have “fashion is the outward expression of belonging to a community”, a theme a returned to several times.

Be prepared

Conference Stuff

Being prepared really helped for me, the more I could practice with an environment or a tool, the more comfortable it was to use during the talk. I made sure that I had my own slide clicker, wifi hotspot, vga adapter, small snack and so on. Even the things I didn’t end up needing that day were helpful to have because I wasn’t stressing out about whether they would be there or not.

Practice, Practice, Practice

Conference Podium

I think I practiced giving the talk something like 14 times before I took the stage on Friday. The number one thing I noticed was that as I practiced I got more and more consistent about what I wanted to say. The first time I practiced my talk all the way through my listener said “That’s great, you have nothing to worry about. Let’s run it one more time” and then I delivered a completely different talk. Because I hadn’t really locked down what I wanted to say I ended up saying completely different things the second time. The more I practiced the more consistent I got.

Another great thing about practicing was that it helped point out where I was being one sided on an issue. A coworked pointed out to me after a practice run “fashion is sensitive”. If someone says that they love or hate your clothes or haircut, that can really hit home. The same is true with code so it’s best to focus on “this is why my team made these decisions” rather than “this technology isn’t well thought out”

Conference Demo setup

I tried to practice every thing I thought could happen the day of the presentation. I setup a demo camp in my living room with a bright light, a presenter display and podium. I tried to get the conditions as similar as they would be the day of the presentation of make sure I was ready. It really helped.

Microphone

I thought there was a chance that there might be a hand held microphone so I wrapped up a remote up in a piece of paper to practice with it. Boy did it throw me! I wanted to talk with my hands so badly but I was stuck with this microphone. But with a little bit of practice I got comfortable with it, and I was relieved I had this experience before I got up on stage in front of an audience.


Conference Venue

The conference itself was a ton of fun and the talk seems to have gone over very well, here’s some nice things that people said:

Quote

Quote2

While it was a lot of work getting ready for this conference, I also had a great time and learned a lot.

IP Addresses as 32 Bit Integers (in Ruby)

I recently came across a web address that looked like http://2130706433 and thought- that can’t be right. I was amazed when I typed it into chrome and saw it resolve to http://127.0.0.1. IP Addresses can be represented as 32 bit integers instead of the typical dotted quad notation.

I decided to code up an implementation in ruby, here’s the result:

  class InvalidIPAddress < StandardError; end;

  class IPAddress
    attr_accessor :address

    def self.valid_ip_address?(address)
      quads = address.split(".")
      return false if quads.length != 4
      quads.each do |q|
        if q.to_i > 255 || q.to_i < 0
          return false
        end
      end
      true
    end

    # Must be initialized with dotted quad ip address form
    def initialize(address)
      raise InvalidIPAddress unless self.class.valid_ip_address?(address)
      @address = address
    end

    # Represents IP Address as a network byte ordered 32 bit integer
    def to_i
      lngip = 0
      quads.each do |q|
        lngip = (lngip << 8) | q
      end
      lngip
    end

    def to_s
      address
    end

    private

    def quads
      @quads ||= address.split(".").collect { |q| q.to_i }
    end
  end

And a quick spec to go along with it

  require 'rspec'

  describe IPAddress do
    let(:local) { IPAddress.new("127.0.0.1") }
    let(:ten) { IPAddress.new("10.10.10.10") }
    describe 'valid ip addresses' do
      it 'is 4 parts separated by .s' do
        expect( IPAddress.valid_ip_address?("127.0.0.1")).to be true
        expect( IPAddress.valid_ip_address?("127.0.1")).to be false
        expect( IPAddress.valid_ip_address?("127,0,0,1")).to be false
      end
      describe 'each part' do
        it 'is 0 or greater' do
          expect( IPAddress.valid_ip_address?("0.0.0.0")).to be true
          expect( IPAddress.valid_ip_address?("10.10.10.10")).to be true
          expect( IPAddress.valid_ip_address?("-1.-1.-1.-1")).to be false
        end
        it 'is 255 or less' do
          expect( IPAddress.valid_ip_address?("255.255.255.255")).to be true
          expect( IPAddress.valid_ip_address?("256.256.256.256")).to be false
          expect( IPAddress.valid_ip_address?("254.254.254.254")).to be true
        end
      end
    end

    describe 'initialization' do
      it 'raises an error if an invalid ip address is passed' do
        expect{ IPAddress.new("127.256.0.-1") }.to raise_error
      end
    end

    describe "to_i" do
      it 'converts the ip address to a 32bit integer' do
        expect(local.to_i).to be == 2130706433
        expect(ten.to_i).to be == 168430090
      end
    end
  end

This code can produce integer representations of IP addresses by using the to_i method. For example:

  IPAddress.new("127.0.0.1").to_i
  #=> 2130706433

Faster Ctrlp.vim for Rails Projects

Ctrlp is a fuzzy finding file matcher that pops up at the bottom of vim when it’s control p is pressed (or your custom binding). It’s a huge timesaver to quickly be able to jump to to app/assets/javascript/views/outfitsView.js by hitting control p then ofsvjs.

One frustration with ctrlp is that it tries to index everything but stops at around 6000 files (by breadth first serach). This can leave files deep in your directory structure unavailable and add frustrating results to the listing. Luckily, it’s easy to fix with per project vimrc files.

  1. Allow per project vimrc’s. In your main .vimrc add:

     " enable per-project .vimrc files
     set exrc
     " Only execute safe per-project vimrc commands
     set secure
    
  2. Add a .vimrc to your project root and add:

     set wildignore+=tags
     set wildignore+=*/tmp/*
     set wildignore+=*/vendor/*
     set wildignore+=*/spec/vcr/*
     set wildignore+=*/public/*
     set wildignore+=*/chef/*
     set wildignore+=*/coverage/*
     set wildignore+=*.png,*.jpg,*.otf,*.woff,*.jpeg,*.orig
    

You might need to adjust the vimrc to suit your project. This has been a huge timesaver for me and resulted in faster file finding and more relevant results.

On final tip, if after adding this you still don’t seem the right results you may need to clear the ctrlp cache. You can do this by going into ~/.cache/ctrlp/ and removing any files that match your directory name.

Faster Vim Searching for Rails Projects

I often need to find a particular string inside of a project. One popular tool to do this is ack and, inside of vim to use the Ack.vim plugin. Until recently I’ve used ack almost everyday. However, ack can be slow, especially when you’re just searching for a string and not a pattern. I’ve found, for my usage The silver searcher to be a faster alternative. You give up the ability to look for patterns and instead must look for static strings, but as I looked through my shell history I found that I’ve used pattern matching once in the last year. Your mileage may vary but for me this is a huge speedup at almost no cost.

Here’s how to get going searching using the silver searcher:

  1. Install The Silver Searcher – Using your favorite package manager:

    brew install the_silver_searcher or apt-get install silversearcher-ag

  2. In your .vimrc tell Ack.vim to use the silver searcher; Add this line:

     let g:ackprg = 'ag --nogroup --nocolor --column'
    
  3. Setup a sensible, .agignore file in the root of your project. Here’s mine for my current Rails project:

     tmp
     log
     chef
     coverage
     perf
     spec/vcr
     vendor/
    

You’ll either want to add .agignore' to your.gitignore` file if you think that this kind of configuration is personal or check it in if you think others on your team would agree with these changes.

At this point everything should continue to work as you expect in vim- Ack.vim will instead use the silver searcher, ignore the file paths you specified, and return a list of files (just faster!).

This has hugely helped speed up searching through the project and made it much easier to find what I’m looking for. In fact because I so rarely look for patterns I’ve gone so far as to alias ack to ag in my zshrc.

Quick Spec Framework in Just a Few Lines of Code

It’s possible to write an rspec like framework in very few lines of ruby. Check out this code that gives you some of the comforts of speccing in just under 40 lines of code:

module QuickTest
  class ::Object
    def toEqual(value)
      self == value
    end
  end

  def let(name, &block)
    value = yield block
    define_method( name ) do
      value
    end
  end

  def subject(&block)
    let(:subject, &block)
  end

  def expect(arg=nil, &block)
    return yield block if block
    arg
  end

  def describe(description, &block)
    puts description
    yield block
  end

  def it(description, &block)
    successful = yield block
    if successful
      puts "\tPASSED: #{description}"
    else
      puts "\tFAILED: #{description}"
    end
  end
end

class Person
  attr_accessor :first_name, :last_name
  def initialize(first_name, last_name)
    @first_name = first_name
    @last_name = last_name
  end

  def full_name
    "#{first_name} #{last_name}"
  end
end

include QuickTest

describe Person do
  let(:first_name) { "Nick" }
  let(:last_name) { "Rowe" }
  subject { Person.new(first_name, last_name) }

  describe "full_name" do
    it "should be the first name then the last name" do
      expect(subject.full_name).toEqual("#{first_name} #{last_name}")
    end
  end
end

Next it would be neat to spec the quick test using quick test.

First Encounter With the Oculus Rift

Nick with the Oculus Rift on

My Oculus Rift arrived today! I probably spent about 2 hours plugged in this evening and here are my initial thoughts:

  • It’s amazing
  • It’s an experience that is best tried to be understood
  • VR sickness is a real thing- after playing half life 2 for about 10 minutes I had to remove the rift and get away for while. I’ll have to come back to it when I’m more conditioned to the experience.
  • The resolution is definitely a limitation- I’m looking forward to higher resolutions to eliminate the screen door effect.
  • The head tracking is excellent for side to side and up and down. I wish it had roll tracking (I see some people have worked around that by duct taping sensors to their rifts)
  • The oculus company clearly is in it with all of their heart. The oculus device comes in a neat little suitcase with foam inserts. It includes all the right cables, carefully packaged up. It has adapters to make sure the oculus works with your setup. The device is well made too. A lot of care went into this and it shows.

What I enjoyed

Titans of space

  • Titans of Space was by far the best. It’s a tour through space in a small shop. It’s meant to give sense of the size of different planets. It blew me away. The motion is gentle, the music is great, and it’s really immersive. I played through once and then handed off the rift and watched someone else go through it. Watching on a laptop screen tells you nothing. You can see “Oh that planet is taking up 70% of the screen” but when you’re inside the rift it’s a totally different feeling. It feels like “WHOAH that planet is HUGE!”

  • Blue Marble is another space experience. You’re in a space station and then exit for an experience of floating by the earth and moon. It’s quiet and serene. It was my first experience and it’s really good.

  • Ibex lets you use your computer as thought it was projected on a giant screen. It was really cool for watching videos. Because the resolution is pretty low on the rift all of the fonts have to be cranked way up.

Oculus Welcome Letter

A Rails API Pattern for Complex Collections

When building APIs in Rails I’ve noticed that controllers get heavier and heavier over time with logic around the paramaters that are being received and the collections that are being built.

I’ve been finding that breaking this collection out into its own class, with its own set of logic and then having a factory method for contructing that collection has huge benefits. These collections are easier to test, easier to optimize, and make changing the logic easier beacuse it’s centralized.

To demonstrate these collections, consider an app that manages an API for recipes. In the recipes controller, we would have code like this:

def index
  @recipe_collection = RecipeCollection.from_params(params)
  render json: @recipe_collection.recipes
end

Then RecipeCollection would looks like this:

class RecipeCollection
  attr_accessor :featured, :time_filter

  # Handles the logic of sorting params into a collection
  def self.from_params(params)
    RecipeCollection.new.tap do |recipe_collection|
      recipe_collection.time_filter = params[:time_filter]
      recipe_collection.featured = params[:featured]
    end
  end

  # Should return an ActiveRecord relation
  def recipes
    query = Recipe.scoped
    query.featured if featured.present?
    query.filter_by_time(time_filter)
  end

  def time_filter
  valid_time_filters.include?( @time_filter ) ? @time_filter : default_time_filter
  end

  def default_time_filter
    "today"
  end

  def valid_time_filters
    %w[ today yesterday this-week this-year ]
  end
end

This, RecipeCollection, class represents a collection of recipes that you would like to build. It can be interrogated about defauls (default_time_filter) and current state (time_filter). It be tested, indepedently of the controller which maps nicely to requirements like “The recipes page, by default, should just include todays recipes”.

I’ve found this pattern of having collections outside of the controller, with methods for defaults and the ability to see the complete state of the collection has really cleaned up recent projects.

As a future improvement, I think the defaults and accessor methods could be cleaned up with some metaprogramming or by making use of a middleware like design pattern.

Rails API Performance Tuning

I’ve been doing performance tuning on a Rails API lately and found that there are a lot of great tools and a few common performance problems that are easy to alleviate.

As long as we’ve had Rails, we’ve also had “Rails can’t scale”. And that’s sad, because it’s not true. While Rails may trade the raw performance of a compiled language, it can provide blistering fast APIs and websites. For the past fortnight I’ve been working on scaling out a high API and how has it been rewarding to watch the graph plunge to sub 100ms response times.

Before diving into performance tuning it’s important to think about what success is. There are always going to be slower parts of the app- tuning is never done. For an API, I think 100ms is a good target. I think below 200ms is acceptable. Github is able to have their API respond in 48ms, which is amazing. It’s also important to figure out which parts of the API are getting used, a 200ms but infrequently accessed action may be acceptable.

Why is it slow?

The first important task when doing performance is figuring out why it’s slow. There are really great tools for finding performance hotspots in Rails.

  • Rack Mini Profiler [Free, Open Source] – Rack Mini Profiler is a small badge that sits at the top of a page that’s rendered as HTML that provides a complete breakdown of how fast the page loads. It’s really easy to include if the app- 1 line if you Gemfile and you’re good to go. It only works well if your rails app also provides an HTML page that accesses the API.

  • Request Log Analyzer [Free, Open Source] – Request log analyzer crawls your logs and tells you what’s being accessed, how long actions are taking to respond and what’s returning errors. I like it because logs don’t lie. You can trust that the numbers are real.

  • Newrelic [Paid] – Newrelic is a performance tool that always watches and monitors your app performance. It does the same thing as Mini Profiler and request log analyzer but provides beautiful graphs and charts. Newrelic makes performance effortless and keeps everyone on the same page. The biggest downside I’ve found is that there isn’t always enough granularity. Sometimes Newrelic will report that a controller is taking 5 seconds but not provide any more informtaion that just “ruby”.

Performance Problems

You don’t want to sacrifice code maintainability so I’ve found that with performing these tweaks it’s important to try them and then, if they don’t work out, take them out. It’s frustrating to add a tweak and then have to revert it but 3 months down the line you’ll be looking at the code, see the tweak and think “That must be important”. Try things, watch the graphs, get feedback from your peers, and make sure your fix is really making a difference. Performant but unmaintainable code doesn’t do anyone any good.

I’ve found a couple of common performance problems:

  • Missing indexes on the database – By far the biggest problem I’ve found and the easiest to fix. This is recognizable by a simple find and slow performance. For example, if Person.find_by_email('test@test.com') is slow, the persons table is probably missing an index on the email field.
  • Tons of queries – If loading the API is causing tons and tons of queries, it might be a good opportunity to use include in the query to pre-load the resource as part of another query.
  • External HTTP calls – Performing and external HTTP call during the request, response cycle has a huge impact on performance. These kind of requests should be eliminated if at all possible (or performed on the frontend, in Javascript). The terrible part of these kinds of requests is that network response time is variable so sometimes these might be fast. It’s not always possible to remove external HTTP calls, but it’s always worth striving for.
  • Not making use of Rails caching – Almost everything in Rails is designed with the idea that caching possible. The Rails Guide on Caching is really helpful for understanding what’s available.
  • Fighting Rails caching – Rails provides really neat Key based caching. Rails will generate a new key every time a model is updated. I’ve found places in code where keys were being generated by hand by mashing different fields together with varying levels of performance and success. The cache is really helpful in ActiveModel serializers.
  • Performing work during the request – Hard work like processing images or analyzing files or calculating complex math should be shuffled off to a background job system like Sidekiq.

My Project manager, Mia, observed that performance tuning is fun because you get to watch the graphs go down and know if you’re making a difference. It’s a great feeling to get APIs to a place where they’re quick. I hope that these tools and common problems help you spot places where your app could be a little faster.