State Of Flux - Home

Generate static pages in Rails

2008-07-14T11:05:00Z

I wanted a nice looking 404, 500 and maintenance pages for my Rails app and I couldn’t serve them from Rails.

My requirements were:

I didn’t want to hand code the pages – I’m using a web framework for a reason!
I wanted to use the application’s layout
I needed that pages to be static so I could serve them when the Rails app was either down or when it had ‘issues’

My solution was to:

create a controller for the purpose of rendering the static pages
tailor your views so you have nice 404, 500 and maintenance pages
modify the layout so that signin and register were not longer present
create a rake task to render the pages and write them out to a file

Items 1 & 2 are just standard Rails stuff – so go nutz young coders.

Item 3 was pretty straight forward, in the layout I put:

123	<% unless controller.controller_name == "errors" %> put your sign-in code here <% end %>

Item 4 was a bit trickier, but this rake task should get you up and running.

  namespace :generate do
    task :pages => :environment do
      require 'action_controller/integration'
      app = ActionController::Integration::Session.new
      app.host! "stateofflux.com"
      [['/errors/error_404', 'public/40 4.html'],
       ['/errors/error_500', 'public/500.html']].each do |url, file|
        begin
          app.get url
          File.open(file, "w") { |f| f.write app.response.body }
        rescue Exception => e
          puts "Could not write file #{e}"
        end
      end
    end
  end

We run the rake task in the development environment then check it in, but you could run it in production if there was production data that you needed to create the page.

activerecord-postgresql-adapter in Rails 2.1

2008-07-13T01:20:00Z

If you get the following message

Please install the postgresql adapter: `gem install activerecord-postgresql-adapter`

It means you don’t have the new ‘pg’ postgresql library installed. This is easily fixed with a bit of

sudo gem install pg

Column editing with Emacs

2008-06-14T04:49:00Z

My OS X lovin’ buddies love to point out how easy it is to manipulate columns in TextMate . But I’m old skool and I still love emacs . So to prove that column editing mode is a cinch I’ve put together my first screencast to demonstrate column editing mode in Emacs. Emacs ships with CUA mode which you need to turn on and then let the good times begin. Enjoy…

Cleaning dirty database data

2008-06-09T05:30:00Z

I have a database with duplicate records in it and I want to know how many records I should have if I clean out the duplicates. Boy is this thing dirty! The dataset I’m working with a mid sized (approximately 2 million records) and it’s a dump from another system. One of the problems is that the data in the dump has been denormalized. The second part of the problem is that some data has been entered multiple times in the source system¹.

Let me give you an example.

blog_example=# \d
              List of relations
 Schema |     Name      | Type  |    Owner    
--------+---------------+-------+-------------
 public | addresses     | table | markmansour
 public | users         | table | markmansour
(2 rows)

If I wanted to count the number of users, this would be straight forward, I’d just:

blog_example=# select count(*) from users;
 count 
-------
     3
(1 row)

But let’s look at the data a bit more closely.

blog_example=# select * from users;
 id | name  
----+-------
  1 | Korny
  2 | Tim
  3 | Korny
(3 rows)

blog_example=# select * from phone_numbers;
 id | user_id |  number  
----+---------+----------
  1 |       1 | 11111111
  2 |       1 | 22222222
  3 |       2 | 33333333
  4 |       3 | 11111111
  5 |       3 | 22222222
(5 rows)

For the purposes of this example I’ll consider a duplicate to be a user with the exacly the same name, phone number and address – the main thing is that there are multiple one-to-many relationships and that there is repetition. In this example the user Korny (users with the id 1 & 3) have the same phone numbers and the same address and should be considered duplicates. In SQL the normal way to group things together is to use the cleverly named “group by” clause, but that doesn’t get us what we’re after². I’d like to see the following:

blog_example=# magic select name, number but put the numbers on the same line
 name  |  number  
-------+----------
 Korny | 11111111, 22222222
 Tim   | 33333333
(2 rows)

This can be done with PostgreSql (if you know how to do this in MySql please let me know!) by creating your own aggregate function . You’ve probably used an aggregate function like MAX or AVG before. I’m after a string aggregation function. You can define one like this:

CREATE AGGREGATE array_accum_text (
    basetype = text,
    sfunc = array_append,
    stype = text[],
    initcond = '{}',
    sortop = >);

This allows related rows to be grouped up, for example:

blog_example=# select u.*, array_to_string(array_accum_text(cast(ph.number as text)), ',') as all_phone_numbers
blog_example=#   from users as u
blog_example=#   inner join phone_numbers as ph on u.id = ph.user_id
blog_example=#   group by u.id, u.name;
 id | name  | all_phone_numbers 
----+-------+-------------------
  3 | Korny | 11111111,22222222
  1 | Korny | 11111111,22222222
  2 | Tim   | 33333333
(3 rows)

To take it a step further we can now group related fields together, but I’ll do it via a view. I want the users id to remain so that when I join the text together it doesn’t collapse all the telephone numbers from all the names even if their user ids are different (this is really hard to explain so I suggest trying it out without a view to see what I mean).

blog_example=# create view extended_users as
blog_example-#   select u.id as user_id,
blog_example-#          u.name as name, 
blog_example-#          array_to_string(array_accum_text(cast(ph.number as text)), ',') as all_phone_numbers
blog_example-#     from users as u
blog_example-#     inner join phone_numbers as ph on u.id = ph.user_id
blog_example-#     group by u.id, u.name;
CREATE VIEW

blog_example=# select name, all_phone_numbers from extended_users
blog_example-#   group by name, all_phone_numbers;
 name  | all_phone_numbers 
-------+-------------------
 Korny | 11111111,22222222
 Tim   | 33333333
(2 rows)

From this query I know that there are only two records once I remove all duplicates. When I ran this over my dataset it took the rows from 2 million down to 1.4 million. That is a lot of redundancy that my users don’t want to see. My next action is going to be to writing some Rails migrations to clean it up :), but that will have to wait for another post.

Footnotes

¹ I want to talk about a technical solution so let ignore the politics of the situation (i.e. let’s presume that I can’t get the data keyed in a better way or have the data delivered in a more normalized format).

² Some SQL:

blog_example=# select u.name, ph.number from users as u
blog_example-#        inner join phone_numbers as ph on u.id = ph.user_id
blog_example-#        group by u.name, ph.number;
 name  |  number  
-------+----------
 Korny | 11111111
 Korny | 22222222
 Tim   | 33333333
(3 rows)

Rubinius comes to Melbourne

2008-03-24T00:43:00Z

Rubinius is a reimplementation of the Ruby interpreter initiated by Evan Phoenix who has recently been hired by Engine Yard. Engine Yard is a hosting company that is positioning itself as the hosting provider of choice for Ruby applications by a variety of developer focused strategies. In an effort to raise their profile and gain credibility in the Ruby community they’ve hired Evan to work full-time on the development of Rubinius. They’ve also hired the excellent Ezra Zygmuntowicz of camping fame which gives them huge cred in the developer community. Not only are they just latching onto established efforts, they are creating new markets by going after the shared hosting crew by developing mod_rubinius . I would guess that the benefits are twofold – they look like nice guys by giving back to the community, but they also create a very easy path for them to upsell shared hosting into VPS. Everyone wins when this kind of effort is put into a business model.

Evan and Eric Hodel are out in Australia for the Rubinius Sprint set up by Marcus Crafter and fellow Engine Yarder Dylan Egan . In a side trip Evan came to Melbourne and spent three hours explaining how Rubinius worked and answering a heap of really interesting technical questions from the Melbourne crew before heading out for a few quiet (5am) drinks.

BarCamp Melbourne 2008

2008-03-21T12:18:00Z

I’ve been a bit absent with my blogging so I’ve got a few retrospective entries for you.

BarCamp Melbourne : 60 people, 25 talks and an amazing day of interdisciplinary cahoots unorganized by Ben Balbo ! What I loved about BarCamp:http://barcamp.org/ was the range of presentations – everything from programming, databases and information modelling through to media centers, industrial relations, social activities and a hell of a lot of fun.

I’d like to give special mention to Paul Fenwick’s hilarious and poignant talk on “An Illustrated History of Failure”. He managed to show why testing costs – and quantified it. It was a showcase of brilliant execution on a topic which would, with an average presenter, make you groan.

Other outstanding talks where Andy G’s “OpenLazlo GUI, jiggy itouch GUI, openmaji” and Brent Snook’s “Lowering Usability Hurdles with the Wii”. Very, very cool stuff. In reality all the talks were excellent and had a lot to offer. I can’t wait to see the what the next BarCamp has to offer.

I was lucky enough to win a copy of The CSS Anthology from SitePoint:http://sitepoint.com/ for my talk on Monkey Patching (which I’d given previously as the Ruby Nuby night a few months ago). The after event drinks were in North Melbourne, with the suds provided by Microsoft

You can keep up with the activities on by checking out the BarCampMelbourne2008 tag on da internets .

Building a site for people with big imaginations

2007-10-03T14:19:00Z

On Friday I presented a talk at Web Directions South conveying my tips on how to build a successful web application. I've embedded the presentation below

I set up the talk by presenting the issues that come with web development, which are usually new product development issues. Basically you don't know what you are building until you build it, so embrace the unknown and plan for it. Even in a sea of change there are things that help you get to market, these include:

Good People - look for proactive people who communicate well

Good Tools - there are lots of great tools that help get things done quickly and cheaply, so use them

Simple Development process - use an agile methodology, they are well thought out but hard to get right

Ship it - if you don't ship code, you don't have a product

Fun - make work fun if you want keep your people and get the most out of them (and you!)

A pretty simple premise really, but it amazing me how easy it is to get this wrong. Usually I see companies get good people and then give them good tools and build a really overweight process around them so it is impossible to push software out of the organization. It's a hard problem and this is why successful web apps (and software in general) actually comes about.

If you did see my talk then please leave a comment, I'd love to hear what you thought of it. But if you didn't I'm also interested in your thoughts on my uploaded presentation.

If anyone has any photos of my presentation I'd love to get a copy - it appears that the official photographer was in the other room during my talk.

I've deleted a boatload of comment spam, so if I've deleted a 'real' comment, please let me know and I'll add it back in.

Let the WebDirections begin...

2007-09-26T08:20:00Z

I’m pumped. I’ve just arrived in Sydney from my one hour flight from Melbourne. I’ve checked in and hooked up to my $30 per day internet connection and I’m putting the finishing touches on my presentation. I haven’t spent much time in Sydney but from what I can see it is a beautiful place. Tonight there is an industry meet and greet which I look forward to downing a few brews and meeting the locals. Sounds awesome.

During the process of writing my talk it has occured to me how much there is to writing software. My talk is targeted at Web Applications, but when all said and done, the same disciplines are required for web work as they are for all other application development. I was going to talk about RedBubble and why it was cool, but it was just a bit too specific – there wasn’t going to be anything concrete to take away. I concluded that a more valuable approach would be to convey (in a compressed timeframe) the a collectin of practices that make a group successful. I assume that the hard skills (HTML, CSS, coding, design) are being worked on because they are the most obvious, but the softer skill and the management practices are the skills that are harder to learn. So, in 50 minutes, I’m going to spew forth a collection of good practices with examples with a compresensive set of references. My hope is that I pique some interest and that the motived ones in the audience will pick up the battern and Go, Go, Go!

If you are at Web Directions please come up and say hello and tell me that you’re reading my blog!

Exceptional Cow!

2007-08-17T01:19:00Z

Surprisingly, we are not perfect. Shocking. Even after our testing, we sometime release code into production with errors. We try to be good boys by not just ignoring these errors but we get the system to log them and send the info to our tech mailing list. In the Rails world, the excellent Exception Notifier plugin, from the prolific Jamis Buck, makes this a breeze. If you’ve got a Rails project and your not using it, then you should. Go on, get it!

But now what? You’ve got the emails but with the constant pressure to get things done it becomes easy for everyone to just think “Oh, I’m a bit busy, I’ll let one of the other guys do it.” But they don’t, because we’re all busy. Introducing, the Exceptional Cow!

The Exceptional Cow (a Tucows squigy toy)

At our end of iteration review, like all good agile shops, we go through what’s good, what could be done better, what still puzzles us and what we are going to do next time (but details on this are for another post). We also have The Exceptional Cow™.

Whoever has the cow is responsible for triaging all incoming exceptions for that iteration. At the end of each iteration The Exceptional Cow is ceremoniously passed to the next bovine herder. As the cow herder, you have the responsibility of examining all incoming exceptions and fixing it if it is a no brainer or writing it up as a bug for someone else to fix if you don’t have the time or if someone else has a much better grasp on the issue. Quite often all exceptions for the week are attacked in the final hours before we close off the iteration as we don’t want to start new functionality at that point.

What it has done for RedbBubble is make our code more robust which means our customers have a better experience. It also makes everyone just a bit more careful and tolerant of exceptions. It is also a great way to get everyone across parts of the code base they wouldn’t normally look at including the systems side of things. The cow is our friend…

Edit: The Tucows developer blog picked up on my post – Thanks!

A full house at Melbourne Ruby Newbie Night

2007-07-08T11:55:00Z

The Melbourne Ruby User’s Group held it’s first ever Nuby night last week. It was somewhat inspired by the RailsConf 2007 RejectConf that Marcus Crafter and I attended earlier in the year with the idea of many short talks presented by many speakings being preferred over a few ‘experts’ presenting on a wider range of issues. It also acted like a gong show – you had a hard limit of 5 minutes and when the gong went off you could imagine a walking stick wrapped around your neck pulling you off the stage. No pressure… Really…

I spoke about the animals of the Ruby world – monkey patching and duck typing. After researching my talk I realized that I only had a cursory knowledge of both topics but preparing for this talk gave me the reason to learn more. I enjoyed the experience and would highly recommend it to anyone who wants to practice their presentation skills.

A big thank you to Pete Yandell for continuing to organize the Melbourne Ruby community and to Marty Andrews for recording and editing the presentations.

You can download my talks

Monkey Patching (Open Classes) – slides and movie
Duck Typing – slides and movie
Photos of the night too

Big props to Justin French also from RedBubble who presented and to Xavier Shay who didn’t present due to time constraints.

A fast grep for Rails

2007-06-17T12:01:00Z

I often want to grep for a word recursively within a directory. That is easy! But quite often I know there are a bunch of directories that I can ignore. For instance, in my rails apps I don’t want to hear about subversion directories and I don’t want to search the vendor tree.

I also don’t want to look at files that are really big – they are usually log files or binaries of some sort that I just don’t care about. So I put together the monster find and grep string that works for me.

find . -path '*/.svn' -prune \
    -o -path '*/vendor' -prune \
    -o -path '*/log' -o -path '*~' -prune \
    -o \( -size -100000c -type f \) \
    -print0 | xargs -0 grep -ne "SEARCH"

Let’s break it down.

find .

– start searching for files recurively, starting from the current directory

-path '*/.svn' -prune

– ignore .svn directories

-o -path '*/vendor'

– OR ignore the vendor directory too.

-o -path '*~'

– OR ignore all the emacs backup files

-o -path '*/log' -prune

– ignore the log directory

-o \( -size -100000c -type f \)

– OR make sure the size is less than 100,000 characters and the type of thing is a file (not a pipe or a director or socket) – this usually gets rid of all non source files

-print0

– tells find to terminate the line with a 0. Which sounds really dull, but you can’t pipe the data (get the output from ‘find’ to ‘grep’) if there is a space in the filename.

– push the data from the command on the left to the command on the right

xargs -0

– take the input from the fine command, terminated with a 0 (the print0 from find) and send it to grep.

grep -ne

– search each file with a regular expression and when a match is found print the line number with the filename.

The easiest way to use this is to put it into a shell file – I’ve put it into s.sh.

My s.sh file looks like this


#!/bin/sh

if  [ ${#1} -gt 0  ]
then
    find . -path '*/.svn' -prune -o -path '*/vendor' -prune -o -path '*/log' -o -path '*~' -prune -o \( -size -100000c -type f \) -print0 | xargs -0 grep -ne  $1
else
    echo "Not enough parameters.  Usage: ./$0 searchstring"
fi

Then I make it executable by

chmod 755 s.sh

If you want to find all of the before filters you would do this

s.sh "before_filter"

and enjoy you super fast find command.