Darwinweb

Convert Syck to Psych YAML format

gabe@websaviour.com (Gabe da Silveira) — Mon, 12 Nov 2012 23:35:00 +0000

If you deal with a lot of YAML in your Ruby code, especially migrating from Ruby 1.8.x to 1.9.2 and 1.9.3, then you may have run into some of the numerous issues with YAML incompatibility. I won’t bore you with a lot of specifics, because most of these bugs are transient, but here are the basic facts:

Syck is the antiquated builtin legacy YAML driver for Ruby.
Psych is the new-fangled modern YAML driver for Ruby.
Ruby 1.9.3 always defaults to Psych.
Ruby 1.9.2 may default to Psych depending on the system state Ruby is compiled.

You can check which engine is active in Ruby 1.9.x like so:

1.9.3p327 :001 > require 'yaml'
 => true 
1.9.3p327 :002 > YAML
 => Psych

Hopefully you are using Psych for everything, but if you have legacy Syck YAML files lying around you could be in for some pain because they are not necessarily compatible. In my case, I had a bunch of i18n translation files emitted with Syck which uses an incompatible escape code structure instead of plain utf-8, thereby rendering the files unreadable by both Psych and humans alike. My solution was a little script utilizing the fact that the engine can be swapped dynamically:

require 'yaml'
require 'fileutils'

if ARGV.empty?
  $stderr.puts "! Must pass one or more filenames to convert from Syck output to Psych output."
  exit 1
end

bad_files = ARGV.select{ |f| ! File.exists?(f) }
if bad_files.any?
  $stderr.puts "! Aborting because the following files do not exist:"
  $stderr.puts bad_files
  exit 1
end

def use_syck
  YAML::ENGINE.yamler = 'syck'
  raise "Oops! Something went horribly wrong." unless YAML == Syck
end

def use_psych
  YAML::ENGINE.yamler = 'psych'
  raise "Oops! Something went horribly wrong." unless YAML == Psych
end

ARGV.each do |filename|
  $stdout.print "Converting #{filename} from Syck to Psych..."

  use_syck
  hash = YAML.load(File.read filename)
  FileUtils.cp filename, "#{filename}.bak"
  use_psych
  File.open(filename, 'w'){ |file| file.write(YAML.dump(hash)) }

  $stdout.puts " done."
end

Converting Hash to OrderedHash in Serialized ActiveRecord Column

gabe@websaviour.com (Gabe da Silveira) — Thu, 01 Nov 2012 04:11:00 +0000

Suppose you have an ActiveRecord class like this:

class Report < ActiveRecord::Base
  serialize :options, Hash
end

And, because you are foolishly still using Ruby 1.8 and doing direct comparisons of the serialized format you need to change it to this:

class Report < ActiveRecord::Base
  serialize :options, ActiveSupport::OrderedHash
end

The problem that you face is that the change will cause any existing rows to start throwing `ActiveRecord::SerializationTypeMismatch`. The solution is to run something like this in a migration or console simultaneous with the deploy:

@tables = %w(reports)
@tables.each do |table|
  puts "============== #{table} =============="

  query = "SELECT id,options FROM `#{table}` ORDER BY id"
  result = ActiveRecord::Base.connection.execute(query)

  result.each do |row|
    id = row[0]
    hash = YAML.load(row[1])

    if hash.is_a?(ActiveSupport::OrderedHash)
      puts "Skipping row ##{id} which is already an OrderedHash"
    else
      ordered_hash = ActiveSupport::OrderedHash.new
      hash.keys.sort{ |a,b| a.to_s <=> b.to_s }.each do |k|
        ordered_hash[k] = hash[k]
      end
      escaped_options = ActiveRecord::Base.sanitize(ordered_hash.to_yaml)
      query = "UPDATE `#{table}` SET options=#{escaped_options} WHERE id=#{id}"
      puts "Running " + query
      ActiveRecord::Base.connection.execute query
    end
  end

  puts ""
end

Drupal vs Rails

gabe@websaviour.com (Gabe da Silveira) — Wed, 03 Oct 2012 19:30:00 +0000

During this discussion on HN, I was surprised to learn that Drupal—the high-level framework I left behind years ago in favor of Rails—has a high-octane elite consulting industry behind it with the same prestige and perhaps even more clout than the Rails community.

This gave me pause, because for me Drupal was always the option for demanding clients with razor thin budgets—The type who waxes on casually about a laundry list of features that sounds like Yahoo’s homepage and then comes with a four-figure budget as if that should amply fund a Facebook clone.

But companies like Acquia are an obvious indicator of just how shallow and murky my end of the pond was when I first formed my opinion on Drupal. What are the true advantages of Drupal vs Rails for elite web consultants?

Contrasts

Drupal has much more common functionality and it’s better integrated. Rails’ functionality is more modular and lower-level.

Drupal theming and layout is easier, getting you to a working site in minutes and styling in general is much less work. Rails is more flexible, being CSS framework agnostic and allowing minimalism or complexity to taste.

Drupal has a heavy data model and rather unique API which allows modules to follow common patterns and work together more naturally. Rails starts with no data model, so you build from scratch but you can tailor to your exact use case without any unused overhead.

Primary Use Cases

Drupal excels in cases where you have huge amounts of content and a wide variety of essentially boilerplate functionality to integrate all over the place. Large content sites like bbc.co.uk or whitehouse.gov would be the poster children for this type of site.

Rails excels where you have a very specific and unique data model and you need to tailor every aspect of the site and UX around that data model. Almost any startup fits the bill here, since it is generally implicit that the backend powering the service is unique. It’s hard to imagine building Github in Drupal, for example.

Summary

In essence Drupal hands you a mountain of functionality that is surprisingly flexible for its sheer volume, but for which the cost of adding new functionality is relatively high. Rails gives you only primitives (maybe with some higher level engine gems that still require a fair amount of glue code to integrate fully), but the cost of building functionality is relatively minimal.

In order for Drupal to make economic sense you need to be able to use a significant portion of that mountain of functionality. In fact you need to use enough of that mountain to cover the additional cost of custom development when there is no applicable module, or UX assumptions diverge from Drupal core’s.

I would summarize the distinction as Portals vs Startups. The archetypical Portal is exactly the kind of site for which Drupal is optimal and Rails would be a money-sink. Likewise, the archetypical Startup would not want to get mired down with Drupal’s prescribed data model when they are trying to iterate rapidly on a concept.

Of course, the vast majority of sites out there are neither as massive as a Portal, or as unique as a Startup. Suddenly the water is murky again, but I think this is still a useful lens with which to make early website technology choices leaving personal biases aside.

The Problem with Rails' Catch-all Route

gabe@websaviour.com (Gabe da Silveira) — Fri, 28 Oct 2011 00:52:00 +0000

Today I am upgrading several hundred lines of Rails routes from the old pre 3.0 syntax to the new Rails 3.0 routing DSL. I do like the new syntax, though it is far from a trivial job as there are a number of subtle edge cases I’ve been dealing with. As I meticulously considered and tested each route I discovered some route matches that I didn’t think should have been covered.

For example:

namespace :services do
  resources :fanships do
    collection { put :create }
  end
end

This was matching:

GET /services/fanships/create

Which was clearly not what was intended.

Okay, I admit this route is an unusual construction in a Rails app. At first glance I thought maybe something funky was going on due to the conflict of the default resourceful create route and the additional one I was creating. So I stripped it down:

namespace :services do
  put 'fanships/create' => 'fanships#create'
end

However, even after verifying that there were no other fanships routes getting in the way (the file is over 500 lines) still a match on GET.

At this point it hit me, the catch-all route:

map.connect ':controller/:action/:id'

This app begin life in Rails 1.1, so the default route has always been there even though the vast majority of the app is served by standard Railsy resources. I never thought much of it, and in fact I’ve relied on it a handful of times over the years. But suddenly doing this exercise it hit me like a ton of bricks that the default route is clobbering all my carefully considered RESTful constraints on URLs. Then I realized that the :controller symbol is traversing slashes as well, thus even my namespaced controllers aren’t safe. As far as I can tell there is no straightforward way to prevent this effect as long as the catch-all is active.

I’m sure this is no revelation to a lot of people—in fact there’s a comment about it in the default routes.rb comments—but I had never thought through the implications. I guess the reason it’s not a bigger problem is because the default resourceful route for show shields the update, create, and destroy methods from a casual GET. But even so, this violates the principle of least surprise in a dangerous way. From now on I’ll be sure to never use the default catch-all route.

In Defense of ORMs

gabe@websaviour.com (Gabe da Silveira) — Thu, 16 Jun 2011 06:00:00 +0000

Laurie Voss just posted an article that ORM is an anti-pattern in which he calls out ActiveRecord (although nothing about ActiveRecord is specifically cited) as doing more harm than good. Some of the flaws he mentions are real issues, but he hand-wavingly dismisses the benefits without any real analysis. Let’s examine the claims:

Some ORM layers will tell you that they “eliminate the need for SQL”. This is a promise I have yet to see delivered.

Which ORM says this? This is just a strawman set up by painting ORM projects as making ridiculous claims. There are quotes around the words, but if you plug them into Google, nothing on the first page is a claim from an ORM, they’re all about eliminating [MS] SQL Server or SQL experts.

Others will more realistically claim that they reduce the need to write SQL but allow you to use it when you need it. For simple models, and early in a project, this is definitely a benefit: you will get up and running faster with ORM, no doubt about it. However, you will be running in the wrong direction.

Now he brings it back to reality and admits you can get up and running faster with ORM, but then claims you’re running in the wrong direction, full stop. There’s no reasoning why you are running just in the wrong direction, just a blind assertion.

“Code Generation” is also mentioned as a benefit, but without going into much detail. “Efficiency is ‘good enough’”, is mentioned here as well but seems out of place; it’s a known tradeoff, not something to be touted as a benefit.

Now getting to the meat of the article.

Inadequate abstraction

The most obvious problem with ORM as an abstraction is that it does not adequately abstract away the implementation details.

This is because there is a fundamental parity mismatch between SQL and imperative code. It’s certainly valid that ORMs are one of the leakiest abstractions imaginable, and therefore they are poor abstractions on the continuum of abstraction quality. That’s a fundamental challenge of using SQL regardless of whether you are using an ORM or not. Writing your SQL by hand does nothing to address that, you still have to convert back and forth between imperative code and data structures to SQL and flat rows of data. The challenge of doing this well and creating a nice interface is the whole reason there are so many different ORMs and why they vary so widely in form and function. If you reject ORMs you’re really just saying that you have a better way of doing it, at least for the project at hand, which may very well be true, or you may wind up digging your very own anti-pattern.

The whole point of an abstraction is that it is supposed to simplify. An abstraction of SQL that requires you to understand SQL anyway is doubling the amount you need to learn: first you need to learn what the SQL you’re trying to run is, then you have to learn the API to get your ORM to write it for you.

The premise of ORM is not that it precludes the need to understand SQL, anyone telling you otherwise is selling snake oil. You’re not learning “twice as much”, the ORM is there to provide a standard way to get your data in and out of an SQL database. It certainly is quicker to learn the primitive SQL driver functions, but all the time and more you saved by not learning the ORM will be spent figuring out how to construct SQL queries and map row data to objects. Again, you might create a more optimized solution for your project, but just as likely you will run into dead-ends and require refactorings because you don’t have the experience of thousands of developer hours with widely varying use-cases behind your homegrown solution.

A defender of ORM will say that this is not true of every project, that not everyone needs to do complicated joins, that ORM is an “80/20” solution, where 80% of users need only 20% of the features of SQL, and that ORM can handle those. All I can say is that in my fifteen years of developing database-backed web applications that has not been true for me. Only at the very beginning of a project can you get away with no joins or naive joins. After that, you need to tune and consolidate queries. Even if 80% of users need only 30% of the features of SQL, then 100% of users have to break your abstraction to get the job done.

Well, in my 27 years of programming (that’s right, I first sat down in front of an AppleSoft BASIC prompt when I was 6), I’ve found that on 34% of projects, we need 58% of the features of 66% of the RDBMS engines available.

But seriously, if you work with ActiveRecord for any amount of time you realize that assertion about the percentage of SQL feature coverage is meaningless. There is no such percentage. I’m hard pressed to think of a feature of SQL that ActiveRecord does not support. It’s true that there are some higher level methods (associations for instance) that do more with less code, but even those can take additional options that tweak the SQL in a variety of ways. Even when you drop down to the more basic building block methods (#select, #where, #group, etc) you still gain the benefits of the ORM in that you can chain them with each other and higher level methods and not have to worry about munging your own SQL strings.

Incorrect abstraction

If your project really does not need any relational data features, then ORM will work perfectly for you, but then you have a different problem: you’re using the wrong datastore.

This is continuing with the straw man that ORMs don’t do joins, but they do. The subsequent line of thought follows directly from this fallacy. Then he comes to:

On the the other hand, if your data is relational, then your object mapping will eventually break down.

This is true in one sense. No ORM can make all possible relations be first class citizens in the library. At some point you have to drop to a lower level. Depending what it is you may need to drop down to the raw data, or maybe an intermediate level. In any case, this is not invalidating the other thousands of queries that for which your app happily makes use of the high-level ORM features. The two can very happily co-exist.

Death by a thousand queries

This leads naturally to another problem of ORM: inefficiency. When you fetch an object, which of its properties (columns in the table) do you need? ORM can’t know, so it gets all of them (or it requires you to say, breaking the abstraction).

So the solution is to force yourself to specify the columns every time without exception? Again, baby right out with the bathwater; it’s no harder to specify in an ORM than it is in raw SQL. Also you may end up shooting yourself in the foot because sometimes RDBMS software has optimizations around SELECT *, and therefore you hurt yourself by removing columns unless you really are shaving a lot of data.

Initially this is not a problem, but when you are fetching a thousand records at a time, fetching 30 columns when you only need 3 becomes a pernicious source of inefficiency.

It’s not pernicious. You just add .select('col1,col2,col3') after you profiled the action and realized this was the bottleneck. Or you could trace the usage of every query, optimize the SELECT clause for each and every case, and endlessly refactor as your application and interface grows and changes. You could also write the whole thing in binary and I bet it would be pretty easy to squeeze out twice the performance in 100x the development time.

Many ORM layers are also notably bad at deducing joins, and will fall back to dozens of individual queries for related objects.

On the other hand if you are joining several relations to a base table, this is exactly what you want to avoid cartesian product explosion. Contrary to the implication, ORM developers do profile and optimize their code. You can often do a better for a specific case, but it’s far from being necessary every time.

The problem, I have discovered with experience, is that there is seldom a single “magic bullet” query that needs to be optimized: the death of database-backed applications is not the efficiency of any one query, but the number of queries. ORM’s lack of context-sensitivity means that it cannot consolidate queries, and must fall back on caching and other mechanisms to attempt to compensate.

The use of “magic bullet” here frames the idea of selective optimization in a negative light, because all programmers know there are no magic bullets. Unfortunately it’s a poor use of metaphor because a bullet refers to a solution not a problem. A more apt platitude would be that premature optimization is always bad. There is more likely more than one problem query in a large application, but there are also likely hundreds of inconsequential boilerplate queries needing no more fine-tuning than an index or two.

In any case, it would be disengenuous to say that ORMs don’t come with performance overhead, but it’s a measured tradeoff. There’s no cliff-dropping performance hit as indicated by the last vaguely damning sentence about using caching to “compensate”, whatever that means.

Solutions: Use Objects

If your data is objects, stop using a relational database. The programming world is currently awash with key-value stores that will allow you to hold elegant, self-contained data structures in huge quantities and access them at lightning speed.

Okay this is a bit offtopic for a rant like this, but I can’t let it slide. Your data “is” not either objects or relational. The choice of data structures and data stores is an abstract representation of concepts that can be interpretted many different ways. It really bothers me whenever someone offhandedly says that there are many types of data that aren’t relational. Quite frankly, I think that’s horseshit. Let me go on record that I think that there is no useful data that can not be made more useful through solid relational modeling as opposed to some object DB dump.

With a relational model you have a far more powerful querying paradigm than you have on any object data store. By its nature, a relational model is meant to slice and dice data in unforeseen ways. Of course, this flexibility comes with a price. The reason for all these new high performance data stores is to commit to a less-flexible data model that allows you to break out of the performance limitations of a strict ACID RDBMS.

The reason people start with a relational database is because it’s the most flexible way for your data to sit. You can pivot your startup to completely different purposes using the same data and all you need to do is change a few indexes around to support your new query structure. When you use some kind of document or object-oriented store, you are making a much firmer assertion about exactly how this data will be used, and what kind of queries and reports you intend to run from it.

The fact is that most applications will never outgrow the capability of a single moderately powered relational DB server. If and when you do, you will have scaling issues that don’t magically go away just because you chose the high-performance datastore dujour. You’ll have a lot of architecting and optimization to do regardless of the path you take eliminating bottlenecks, so I think it’s entirely justified to slap in an SQL database on day one with the realization that you may have to change it later.

Solutions: Use SQL in the Model

However, remember that the job of your model layer is not to represent objects but to answer questions. Provide an API that answers the questions your application has, as simply and efficiently as possible.

I would add changing state to this, but fundamentally I agree with the philosophy. Given the fact that interacting with a relational database can be messy, I think it’s very smart if you can keep the logic entirely contained in the model, and therefore I could accept that keeping ActiveRecord primitives out of your controllers and views might be a good idea. In practice I only work this way for more complex types of “questions” that I’m asking the model, and Rails sort of blurs the lines by defining a bunch of methods, which could theoretically be replaced if you ripped out ActiveRecord and replaced your backend with something completely different.

OO is itself an abstraction, a beautiful and hugely flexible one, but relational data is one of its boundaries, and pretending objects can do something they can’t is the fundamental, root problem in all ORM.

This closing statement implies a false belief about the design criteria and intention of ORM systems. It seems like a case of misattributing the beliefs of newbies who wish SQL didn’t exist to ORM developers who clearly have a deep understanding of SQL. In reality ORMs are mostly designed for experts. I haven’t used a lot of other ORMs extensively, so I can’t comment on the design criteria or optimal usage of something like SQLAlchemy, but for ActiveRecord and probably a lot of ORMs there are some clear wins:

Automatically map database types to the equivalent programming language types and vice-versa. If you’re managing your own SQL then you’re going to waste a lot of time figuring this out. It’s one place where sensible defaults go a long way. Why debug the edge cases yourself?
Greatly reduce error-prone SQL string munging. Assuming you have your pants on straight and don’t open up any injection holes, you still have to have complex string manipulation logic to compile conditions and joins, or else a bunch of very similar but different queries that are difficult to read and hard to refactor.
Composability of queries. With a proper relational algebra engine, you can programmatically build up a query by chaining clause methods together. Doing so allows you to compose many complex queries and keep the code in the logical place without the design being driven by how you are organizing the SQL itself. An example would be if you have the concept of Authors who might have a “new” flag, and Posts that might have a “rant” flag. It’s clear that the definition of those flags belongs in the respective models, but what if you want to query both at once? In the case where you manually craft your SQL you have to put it somewhere, maybe in one model or the other, or somewhere else. You also have to decide if you are going to munge a bunch of conditions together in one query or split them out. Throw something like an advanced search form in the app and this becomes a minefield of niggling details.
Simplify the simple stuff. There are tons of boilerplate queries in any sizable app. An ORM gives you a concise way to call them and get a standard default mapping of fields to attributes.
Facilitate higher level patterns. The concepts of belongs_to and has_many in ActiveRecord are a simple encapsulation of very common patterns that come up again and again in relational databases. Through a series of simple options it allows you specify and execute a wide variety of queries for counting, instantiating, or loading multiple collections. They can be chained with the lower level functions, and always output correct and straightforward SQL. Sure they can’t do everything, but they certainly are capable of handling the majority of queries in a typical app quite elegantly.

The bottom line is that the concept of an ORM is a catchall for a range of projects that implement common patterns for using SQL in an object-oriented system. An ORM is nothing more than a codified set of patterns. Because of the nature of the relational model, there are many many different approaches one can take. To dismiss ORMs entirely is to dismiss SQL as well. You might claim that you don’t use an ORM in your code, but in effect you are just implementing your own idea of an ORM. If you are just peppering queries around your model then it might be a very lightweight ORM, but the fact is you are still Mapping Objects to Relations and vice-versa.

Aws::S3 Gem Bucket Does Not Exist Bug

gabe@websaviour.com (Gabe da Silveira) — Thu, 10 Jun 2010 06:27:00 +0000

I just spent two hours debugging the most banal bug. It eventually reduced to the following wtf moment:

S3Object.store('a_key', 'some_content', 'a_bucket')
=> success
S3Object.find('a_key', 'a_bucket')
=> AWS::S3::NoSuchBucket: The specified bucket does not exist
    from .../vendor/gems/aws-s3-0.6.2/lib/aws/s3/error.rb:38:in `raise'
    from .../vendor/gems/aws-s3-0.6.2/lib/aws/s3/base.rb:76:in `request'
    from .../vendor/gems/aws-s3-0.6.2/lib/aws/s3/base.rb:92:in `get'
    from .../vendor/gems/aws-s3-0.6.2/lib/aws/s3/bucket.rb:104:in `find'
    from .../vendor/gems/aws-s3-0.6.2/lib/aws/s3/object.rb:172:in `find'
    from (irb):7

Okay, that makes no sense. It works in a vanilla irb. I smell a monkeypatch.

Sure enough, it turns out the Contacts gem and the AWS-S3 gem both define a Hash#to_query_string method. Here’s the kicker: both of those gems only call this method one time!

This is why monkeypatching should be considered harmful. I just wasted two hours of my life so someone could call options.to_query_string instead of to_query_string(options).

Configuring MySQL for utf8 under homebrew

gabe@websaviour.com (Gabe da Silveira) — Sat, 08 May 2010 04:13:00 +0000

Man, every time I set up mysql it takes me half an hour to google up all the documentation of how to make it just work™ with utf-8 instead of the god forsaken latin1 default. So here it is once and for all.

Server my.cnf should contain:

[mysqld]
collation_server=utf8_general_ci
character_set_server=utf8

The location of my.cnf varies depending on how you install it. dev.mysql.com has a nice summary of how my.cnf is looked up.

Since I use homebrew my location is /usr/local/var/mysql/my.cnf. Binary installations differ.

Server config variables are described under Server System Variables

Client configuration goes in ~/.my.cnf:

[client]
default-character-set=utf8

Client config variables are described under mysql Options

You can check the status of these with SHOW VARIABLES

Finally, as a reminder for homebrew users, if you use the com.mysql.mysqld launchd plist that comes with the homebrew formula then you will want to start and stop the server via launchctl:

launchctl stop com.mysql.mysqld
...
launchctl start com.mysql.mysqld

The Case for Git Rebase

gabe@websaviour.com (Gabe da Silveira) — Sun, 25 Apr 2010 06:18:00 +0000

When you first sit down with git they tell you to watch out for rebase. “Git is fast. Git is great. Git gets merging right. If you screw up git reflog has your back. But watch out for rebase, as long as you avoid rebase you’ll never get in too far over your head.”

Okay, I can see the wisdom in that.

In fact for the first year I avoided rebase almost entirely. I read Rebase Considered Harmful early on and it reaffirmed my choice. But as I came to understand git’s internals, and as new descriptions of rebase came online, the distinct feeling that I was missing something started creeping in.

Ironically, what pushed me to really grok rebase was the need to perform a surgical 3-way rebase for a series of commits that were drastically misapplied. Once I understood how to do that rebase was layed clear to me. Rebase is at its essence simply takes a series of commits as if they were individual patches, and applies them at a different point in history. The confusion and opacity of rebase comes in large part from the fact that the range of commits and “base” commits are determined somewhat magically by the branch names that are passed to git-rebase.

Once I understood rebase, I started using it more, but I still held back from suggesting the use of rebase to the rest of my team; the clean history was nice but I didn’t see a compelling enough advantage to become an advocate for a rebase-based workflow.

Things have changed.

A private team workflow

Git was built to manage Linux kernel development, so it’s no surprise that discussion of rebase tends to be focused on open source workflows. In open source once you push a commit to a public repo, you don’t know who else has it, and rebasing public commits will lead to dangerous cascading effects. In a private repo it’s still a good rule of thumb not to rebase pushed commits, but with small teams you can bend the rules just a hair to optimize the history linearity.

The advice I’m proposing to my team is to:

Always use git pull --rebase
Rebase topic branches against master (or the current base branch) before pushing for the first time
Rebase topic branches just before merging and deleting them (and let other people know the branch is officially dead so they don’t keep committing to their local copy)

Why go through the trouble of all this rebasing? Won’t we be losing history? Well yes, rebase vs merge is always a tradeoff. For a long time I thought it was basically a wash: readability in exchange for precise history. However as I came to understand the tradeoffs things kept shifting towards rebase.

Readability

If you are following good agile practices and keeping your stories really short, ideally your topic branches are short and sweet and they all get created and merged within a day or two, right? In that case a few merge commits are not really a burden, and you can easily parse out the exact history of who did what, when. Right, but in the real world some branches end up sticking around longer for various reasons. It doesn’t take long to reach a threshold where the full history becomes unparseable by the human brain. Here is a recent example… this is with just 4 developers!

Once you reach that point you’ve lost the benefit of having full history, and all those merge commits are just useless noise. And it gets worse.

Bug Locality

All else being equal, the readability is not enough to tip me in favor of rebase. However when it comes to debugging, a linear rebased history is your friend as well.

Often times the combination of topic branches result in a conflict. Merging seems simpler because you resolve all the conflicts at once. With rebase you have to fix each conflict as it occurs commit-by-commit. However with each individual commit it’s easier to resolve because the commit is (hopefully) more focused than the entire branch, so the resolution is done in the same context as the original commit was. You think to yourself, “what was the purpose of this commit, and how should it be different given the wider changes that occurred on the base branch?”

With either workflow you have the possibility of bugs. Either because you flubbed the merge or, worse, because of some subtle interaction that you may not discover until much later. This is where bug locality comes in.

If you’re using git-bisect months after the fact, what commit will appear to have caused the bug? In the case of merge it’s going to be a huge commit combining two branches with many changes. This is fairly likely to be completely useless. However, if you have always rebased and have a perfectly linear history, you will always be able to trace it back to a single logical commit. This is the kind of thing that is hard to appreciate until you’ve actually seen git-bisect turn up useless a few times.

Okay, so I admit it’s not always feasible (or even worth it) to maintain a perfectly linear history. A few merge commits here and there aren’t going to hurt anybody. This is one reason I kept my rebasing to myself for a long time. However if you rebase at all then there is a third downside with merging.

Merging is Viral

If you merge all the time, you can find yourself in situations where you’d like to rebase but can’t for practical reasons. This isn’t really a weakness in git so much as the fact that rebasing is easier the closer it’s done to the actual commits. Rebasing your own commits a-la git pull --rebase is more or less the same difficulty as merging (most of the time).

However if you go back to rebase a sequence that has a bunch of merge commits in it, git-rebase will not be able to make use of any conflict resolution done in those merges. This is because the individual commits are replayed one by one in temporal order, which means conflicts that were resolved in later merges have to be re-resolved piece by piece.

Consider rebasing a long-lived topic branch back to master:

On the left you have a topic branch worked on by two people who were regularly merging. On the right you see the master branch which had it’s own line of development going on simultaneously. Now when it comes time to merge this branch down to master, you want to rebase and then delete it. The only problem is that as each commit is replayed, you hit every conflict that originally occurred and was resolved in those merge commits, except now these changes are potentially ancient history, and even if you were one to original do the merge you may not clearly remember the context of each individual commit.

If the two developers had done git pull --rebase every time, they would have resolved conflicts locally so that the later rebase to master would not have any old conflicts to resolve. In this case the conflicts were gnarly enough that rebasing was not practical. Once that happens then rebasing becomes impossible for any branch containing this sequence.

Of course eventually you expect to merge everything back to master and you get a clean slate, but the point is that little merges require bigger and bigger merges as a topic branch grows. Since you don’t necessarily know the life cycle of a topic branch when you start, keeping history clean is a smart hedge.

Conclusion

It took a couple years of daily git use on private projects but I’ve now come to believe that the benefits of a clean linear history outweigh the benefits of a perfect historical record. git-rebase maintains the commit dates also, so you can infer a good deal about the original history. An original history may give me a clue about what a developer was thinking at the time, but this is not necessarily of greater benefit than knowing in a clear order what changes were applied to the codebase. In the end a more powerful git-bisect is the trump card that puts me firmly in the camp of rebase, at least for private projects.

Testing ThinkingSphinx with Test::Unit and Transactional Fixtures

gabe@websaviour.com (Gabe da Silveira) — Sat, 13 Feb 2010 09:25:00 +0000

I first started using Sphinx about a year ago. At The Auteurs we use it to solve some thorny localization and filtering issues, and we have been pushing its limits almost since the beginning. Our film index in particular is quite complex, hitting ten tables and defining two dozen attributes. I’ve been impressed with how flexible it is with a pretty basic set of primitives. But one thing that never had an obvious solution was testing. I just finished the third revision on our test harnesses, and I thought I’d share my approach because it runs a bit contrary to the current zeitgest in Ruby testing circles.

Integration vs Unit Testing

I’m a big believer in integration testing, and when we first setup Sphinx we really wanted to test it end-to-end. Our first two harnesses reflected this approach. We ended up with a fairly elegant method to start and stop sphinx on classes that needed it. This gave us good test coverage, but eventually cracks started to show. We already needed to speed up our tests as I outlined a couple weeks ago, and integrated Sphinx support was one of the worst offenders:

Startup time for sphinx is on the order of a couple seconds, this adds up over multiple tests
Indexing time is also significant, and it adds up even faster if you reindex for every individual test
We’re regularly adding new models to be indexed by Sphinx which is magnifying these sources of overhead
Our indexes are getting more complex, which means exercising them well at the integration level is too slow
Rails foxy fixtures create sparse ids which Sphinx is not in love with
Even worse, if the ID overflows Sphinx internal key size (which I think depends on whether it is compiled as 32 or 64-bit) then those records are silently dropped.
We rely on transactional fixtures for performance reasons, which are not efficient to turn off on a case by case basis to solve the previous problems

Designing Sphinx Tests for Performance

Clearly Sphinx can be quite complex and good test coverage is critical. But what constitutes ‘good’? The meat of Sphinx is simply querying some efficiently indexed data. Verifying the results that ThinkingSphinx#search returns allows you to test that everything between the index definition and the query is working as expected. There are a lot of moving parts under there, so it’s reassuring to have a solid sanity check that your indexes are doing what you think they’re doing.

Given that Sphinx is a high performance fulltext search, there’s no reason you can’t run a lot of tests quickly. I decided that for the health of our test suite and testing discipline to stub out Sphinx in all our tests, and move the Sphinx tests into its own suite of highly efficient tests covering the actual indexes. In the end we lose a bit of regression coverage over the actual Sphinx calls in the application, but that is a comparatively small surface area for potential problems in exchange to encouraging better testing of the complex part, plus it’s not likely to be subtly broken by some distant change and if breakage does make it to production exception notification has our back.

Step One: New Test Suite

In order to isolate these tests from fixtures and minimize startup overhead, I moved all Sphinx tests to test/sphinx and wrote this Rake task:

namespace :test do
  task :sphinx => 'ts:config_test' do
    puts "! Starting Sphinx by rake"
    silence_stream(STDOUT){ Rake::Task["thinking_sphinx:start"].invoke }
    begin
      Rake::Task["test:sphinx_without_daemon"].invoke
    ensure
      puts "! Stopping Sphinx by rake"
      silence_stream(STDOUT){ Rake::Task["thinking_sphinx:stop"].invoke }
    end
  end

  Rake::TestTask.new(:sphinx_without_daemon) do |t|
     t.libs << "test"
     t.pattern = 'test/sphinx/*'
     t.verbose = true
   end
end

namespace :ts do
  task :config_test do
    ENV["RAILS_ENV"] = 'test' # This is a horrible sin against humanity, but it ensures that A) the test config is
                              # always up to date, B) doesn't require devs remembering to add RAILS_ENV=test when
                              # they call rake and C) doesn't incur a penalty of shelling out to 2nd env
    Rake::Task["environment"].invoke # This must be run after the test environment (as opposed to as a prereq) is
                                     # forced or sphinx gets the wrong env.
    puts "! Configuring Sphinx by rake"
    silence_stream(STDOUT){ Rake::Task["thinking_sphinx:configure"].invoke }
  end
end

task "test" => ["test:sphinx"]

The main task here is test:sphinx which first ensures that the test config is up to date (by means of some fairly brutal methods as noted in the comments) and starts and stops Sphinx before running the actual tests.

Step Two: SphinxTestCase

To finish the optimization and make the test-writing experience smooth, a custom test case was required. The following has 3 purposes:

Start and stop Sphinx, but only if it’s not already started globally by Rake
TRUNCATE tables this test uses so that left over fixture remains get cleared and ids start at 1 again
Instantiate an set of test data, but only once per test class

The result:

class SphinxTestCase < ActiveSupport::TestCase
  cattr_accessor :sphinx_started_within_test_case

  class_inheritable_accessor :sphinx_tables_to_index

  def self.indexes_tables(*args)
    self.sphinx_tables_to_index = args.map(&:to_s).map(&:tableize)
  end

  # We use inherited hook to define this only on the concrete test class otherwise the method gets called twice.
  def self.inherited(subclass)
    def subclass.suite(*args)
      mysuite = super
      def mysuite.run(*args)
        unless ThinkingSphinx.sphinx_running? # When running via rake we only start sphinxd once
          puts "! Starting Sphinx on per-test basis" # If running by rake and somehow sphinx doesn't start we should know about it
          silence_stream(STDOUT){ ThinkingSphinxTestHelper.start! }
          SphinxTestCase.sphinx_started_within_test_case = true
        end

        super

        if SphinxTestCase.sphinx_started_within_test_case
          print "\n! Stopping Sphinx on per-test basis"
          silence_stream(STDOUT){ ThinkingSphinxTestHelper.stop! }
        end
      end
      mysuite
    end
  end

  # Instance variables set here are automatically set on each individual test instance
  def self.setup_database(&block)
    @database_setup_block = block
  end

  def setup_fixtures
    unless sphinx_tables_to_index.present?
      raise "Tables to be cleared must be defined using indexes_tables on the test class"
    end

    @@sphinx_test_case_already_loaded ||= {}
    unless @@sphinx_test_case_already_loaded[self.class]
      sphinx_tables_to_index.each do |table|
        ActiveRecord::Base.connection.execute("TRUNCATE #{table}")
      end
      db_setup_block = self.class.instance_variable_get(:@database_setup_block)
      raise "setup_database was not called for #{self.class}" unless db_setup_block
      db_setup_block.call
      ThinkingSphinxTestHelper.index!
      @@sphinx_test_case_already_loaded[self.class] = true
    end

    self.class.instance_variables.each do |ivar|
      self.instance_variable_set(ivar, self.class.instance_variable_get(ivar))
    end
  end

  def teardown_fixtures
    # Do nothing.
  end
end

Test class methods

There are two class methods defined indexes_tables and setup_database that a SphinxTestCase uses to define its environment in lieu of fixtures. These methods respectively take a list of symbols ala standard Rails class methods and a block inserting a set of database rows.

Once-per-class hook

The self.inherited block is a bit hard to parse, but it’s essentially just a way to run something once per test class, similar to RSpec’s before(:all). It has to be called in the inherited callback because otherwise the super call ends up recursing and the code executes twice.

This idea was adapted from work by James Adam, thanks James!

Override fixtures setup

The setup_fixtures and teardown_fixtures are methods defined by ActiveSupport::TestCase. They are completely overriden here since there’s nothing efficient that can be done with transactional fixtures enabled anyway. Instead we just verify that some tables are specified, truncate them, setup the data, and index it. But this only happens once per test class, therefore we have to copy over the instance variables to provide the usual semantics.

Isn’t this data sharing A Bad Thing?

This design means that adding or deleting data within tests will be error prone since it can affect other tests. However for Sphinx you would need to reindex anyway, and because Sphinx is all about querying full datasets, it made sense to me to set up one big set of test data and then right multiple tests to query against it. The worst case is that you simply need to create a new test class to setup a new context, which personally is no less distasteful than contexts nested 3 or 4 levels deep.

Why not just truncate all tables?

That’s not a bad idea actually. Forgetting to specify tables tends to lead to bizarre duplicate key errors. However in our database we have 150 tables, so I decided to require selectivity. YMMV.

Wheres the code?

This seemed like too specific a use-case to release in any formal fashion, but let me know if you disagree.

Step 3: Write Your Tests

Here’s an example using FactoryGirl and Shoulda:

class UserSphinxTest < SphinxTestCase
  indexes_tables :users, :emails

  setup_database do
    @user  = Factory(:user, :first_name => "Will", :last_name => "Fulignoranz")
    @user2 = Factory(:user, :first_name => "Farmer", :last_name => "William")
    @user3 = Factory(:user, :first_name => "Bubby", :last_name => "")
  end

  should "find will" do
    results = User.search('will')
    assert results.include?(@user)
    assert results.include?(@user2)
    assert ! results.include?(@user3)
  end

  should "find ignorance" do
    assert_equal @user, User.search('fulignoranz').first
  end

  should "not find the drummer" do
    assert_equal 0, User.search('friendly fred').size
  end
end

Results

19 tests, 36 assertions, finishes in 7 seconds, full test suite 20 seconds faster with +25 assertions and marginal cost for additional Sphinx tests is now negligible.

Questions?

Benchmarking Rake Tasks and Trivial Rails Testing Optimizations

gabe@websaviour.com (Gabe da Silveira) — Fri, 29 Jan 2010 19:00:00 +0000

I’ve been thinking a lot about testing recently. Perhaps the most inconvenient truth about automated testing is that even though it’s many orders of magnitude faster than manual testing, it’s not infinitely fast, and the marginal cost does add up. This is especially true in slower, interpreted languages like Ruby.

Recently our test suite at The Auteurs has been approaching 15 minutes on a fast MacBook Pro. The obvious solution to this is continuous integration, but I believe there is lower-hanging fruit. Being able to run the full suite (or at least the majority of it) fast means a lot to the agility of the team. There is always room to optimize the tests themselves, but I’ll leave that topic for another post. What was of interest to me is that the default Rails testing tasks seemed to be getting quite slow on a code base of our size (30k lines).

How to attack the problem?

Benchmarking Rake

Rake 0.8.7 doesn’t come with any built-in benchmarking options, but hooking in proved extremely straightforward (thank you Ruby!):

require 'benchmark'

class Rake::Task
  def execute_with_benchmark(*args)
    bench = Benchmark.measure do
      execute_without_benchmark(*args)
    end
    puts "  #{name} --> #{bench}"
  end
  alias_method_chain :execute, :benchmark
end

That produces output like:

environment -->   8.580000   1.390000   9.970000 ( 10.408723)
db:abort_if_pending_migrations -->   2.600000   0.040000   2.640000 (  2.662062)
db:test:purge -->   0.010000   0.000000   0.010000 (  0.146904)
db:schema:load -->   0.810000   0.070000   0.880000 ( 49.263301)
db:test:load -->   0.820000   0.070000   0.890000 ( 49.274854)
db:test:prepare -->   0.830000   0.070000   0.900000 ( 49.422403)
test:units -->   0.050000   0.080000 103.930000 (118.107608)

It’s important to note here that this output doesn’t explicitly communicate nested tasks, and thus you can’t naively sum the totals. Rake mostly operates using prerequisites so that things run one at a time without being nested, but in this case db:test:prepare invokes db:test:load which invokes db:schema:load, thus the former includes the latter execution times.

Also I benchmarked a bunch of things in test_helper.rb, but most significant ended up being the environment:

require 'benchmark'

bench = Benchmark.measure do
  require File.expand_path(File.dirname(__FILE__) + "/../config/environment")
end
puts "  Require Rails Env --> #{bench}"

which ended up being around 10 seconds just like in the rake task:

Require Rails Env -->   8.270000   1.820000  10.090000 ( 10.205786)

Rolling Up Testing Tasks Into One

If you aren’t thinking carefully you might assume that rake loading the environment is basically free since the tests need it anyway, but remember, the rake process is just running commands in a shell, it’s not invoking the tests in the same Ruby process. Therefore each block of tests (ie. unit, functional, integration) needs to startup the environment from scratch. That works fine for small applications, but as they grow the overhead becomes significant. In our case we even have a couple extra blocks of tests beyond the Rails defaults, all of which adds up to significant overhead.

However it’s easy to write a task that runs all the tests in one block:

namespace :test do
  Rake::TestTask.new(:fast) do |t|
    files = FileList["test/unit/**/*_test.rb",
      "test/functional/**/*_test.rb", "test/integration/**/*_test.rb"]

    t.libs << 'test'
    t.verbose = true
    t.test_files = files
  end
  Rake::Task['test:fast'].comment =
    "Runs unit/functional/integration tests in a single block"

The potential issue with this is that you lose the namespacing of the tests, which could lead to test name collisions, or there may be other types of environment collisions depending on the specifics of your app. However in our case this worked right out of the box.

Drop Test Preparations

The other thing you’ll notice about that rake task if you compare to the ones defined in railties/lib/tasks/testing.rake is that I’ve dropped the prerequisite db:test:prepare from the definition. That means that the task won’t automatically reload the database schema, and it won’t warn you if migrations haven’t been run. It also means the environment isn’t required either, yielding a savings of over a minute. Obviously this is highly dependent on the number of tables in the schema. In our case we have 130 which doesn’t seem like a tremendous amount—each table is adding over a third of a second of overhead. If our unit tests were optimized a bit this overhead could end up dominating the execution time.

The downside here is that you are basically on the hook for calling db:migrate and db:test:prepare, but the number of times we need to call those is dwarfed by frequency of test runs. It does require a bit of mental overhead to recognize test failures caused by a bad schema, but the performance gains are well worth it for a large app.

Isolate Slow Tests

Now that the rake overhead is removed, the next logical step is to optimize individual tests. So grab the test_benchmark gem and profile your tests. In our case there were a few super slow tests dominating the runtime profile. These tests are important, but not necessarily in proportion to their runtime. My solution was to cordon off the slow tests in their own directory:

Rake::TestTask.new(:slow) do |t|
  t.libs << 'test'
  t.verbose = true
  t.pattern = 'test/slow/**/*_test.rb'
end
Rake::Task['test:slow'].comment = "Runs the slow tests"

Just to be clear, I recognize that having a set of isolated tests like this is a code smell. In order to be fully comfortable with this I need to have continuous integration setup running these tests so that any developer pushing code will be notified within 20-30 minutes if one of these tests is failing. The other mitigator is that these tests should not be regression tests of core functionality. That’s a judgement call, but with experience it’s not hard to classify some tests as more important than others.

Roll It In

I don’t want to mess around with the built-in Rails tasks because that is confusing to new developers. However I do want test:fast to be the default task because that’s going to be our bread and butter. So in the end I came up with these changes:

Rake::Task[:default].clear
task :default => "test:fast"

task "test" => ["test:slow"]

The last bit ensures that rake test will first run test:slow which means that we now have the best of both worlds at our finger tips. rake will run the fast tests, and rake test will run the standard Rails test blocks, preceded up the slow tests.

I chose to move 4 out of 250 test classes to the slow/ set. Testing the total runtime from the shell using the time utility yields:

sh> time(rake test)
real 11m54.247s
user 8m56.002s
sys  0m30.125s

sh> time(rake)
real 8m1.043s
user 6m36.532s
sys  0m16.860s

So we went from 12 minutes down to 8 minutes by dropping less than 2% of the test classes. Not a bad first effort. Next up: optimization techniques for Rails test suites.