Peer Assembly Blog

Escaping Filename Spaces in Paperclip Attachment URLs

2011-10-14T00:00:00+01:00

As I’ve mentioned before we generally use Paperclip when we need to support file uploads. However, there are a couple of edge cases which Paperclip, out of the box, doesn’t solve. One of these is the correct URL escaping of uploaded filenames (at least as of version 2.3.15).

The Problem

Imagine you have a model with a Paperclip “picture” attachment…

has_attached_file :picture, :styles => { :thumbnail => "300x200>" },
  :storage => :s3, :s3_credentials => "#{Rails.root}/config/s3.yml",
  :default_url => "/images/default_picture_:style.png",
  :path => "pictures/person/:id/:style_:filename"

This would be accessed in your view using…

@person.picture.url(:thumbnail)

All looks good. But Paperclip won’t URL escape the URL it generates for this attachment. The first time people generally notice this is when the uploaded filename contains spaces. If you upload a file with a name like “covent garden.png”, Paperclip will generate the invalid URL http://s3.amazonaws.com/bucket/pictures/person/141/thumb_convent garden.png

Drat that pesky space.

The Solution

There are two ways to solve this problem

Option 1: Rename the File on Upload

This option is generally the most popular. It involves writing a PaperClip before_post_process filter which renames the file after it has been uploaded. This new name can be URL safe. So even though the subsequent URLs Paperclip generates are not escaped it isn’t a problem.

There are several articles on this mechanism already so I won’t go into the implementation details here.

Make URL friendly filenames in paperclip attachments

I’ve found there are several problems with this mechanism. The most obvious is that it needs to be in place before any images are uploaded.

Option 2: URL Escape the Filename When URL is Generated

My preferred option is to write a new Paperclip interpolation.

Create a new file in your ruby project..

config/initializers/paperclip_interpolates.rb

With the following content…

# Paperclip by default does not escape the filenames. This interpolated
# variables lets you access the URL escaped version of the filename.
# This should be used in the :url option of has_attached_file.
# For example...
#
# has_attached_file :picture, :styles => { :thumbnail => "300x200>" },
#   :path => "pictures/person/:id/:style_:filename",
#   :url => '/pictures/person/:id/:style_:escaped_filename'
#
# Note, this solution is not suitable for S3 based attachments.
#
Paperclip.interpolates('escaped_filename') do |attachment, style|
  "#{CGI.escape(basename(attachment, style))}.#{CGI.escape(extension(attachment, style))}"
end

# To escape the S3 URL takes a little more work. In this scenerio the url
# option should be...
#
# has_attached_file :picture, :styles => { :thumbnail => "300x200>" },
#   :storage => :s3, :s3_credentials => "#{Rails.root}/config/s3.yml",
#   :default_url => "/images/default_picture_:style.png",
#   :url => ':s3_escaped_path_url',
#   :path => "pictures/person/:id/:style_:filename"
#
# Note, paperclip's S3 storage component will ignore url options which don't start with ':s3'
# and end with 'url'.
#
Paperclip.interpolates('s3_escaped_path_url') do |attachment, style|
  original = "#{basename(attachment, style)}.#{extension(attachment, style)}"
  escaped = "#{CGI.escape(basename(attachment, style))}.#{CGI.escape(extension(attachment, style))}"
  
  "http://s3.amazonaws.com/#{attachment.bucket_name}/#{attachment.path(style).gsub(original, escaped).gsub(%r{^/}, "")}"
end

Now update the model’s attachment to have an additional :url option which uses our new s3_escaped_path_url interpolation…

has_attached_file :picture, :styles => { :thumbnail => "300x200>" },
  :storage => :s3, :s3_credentials => "#{Rails.root}/config/s3.yml",
  :default_url => "/images/default_picture_:style.png",
  :url => ':s3_escaped_path_url',
  :path => "pictures/person/:id/:style_:filename"

That’s all the changes you need to make. No file renaming, now all the URLs for that attachment are generated with the correct escaping.

Downloading Device UDIDs from Apple Developer Portal

2011-09-30T00:00:00+01:00

Apple allows an iOS developer to associate up to 100 devices with a developer account, and use those devices for testing Ad-Hoc builds. Although there’s an option to remove devices, they continue to count towards the 100 device limit. That is, until you renew your developer account – then you get a message that you can delete devices and have them no longer apply against the limit. In past years, I’ve tried to carefully prune the list, but inevitably I’ve overlooked a bunch of devices and once you start adding new devices, you can no longer delete old ones.

This year, I have a new plan. I’m going to delete all devices, except for my own, and then gradually re-add client devices as they’re needed. The developer portal has a useful option to upload a text file containing a list of devices but unfortunately doesn’t have an option to download the existing devices. Using the excellent mechanize gem, I wrote a short script to scrape the UDIDs from the developer portal and export them to a local file. Here’s how you use it:

$ ruby dl_devices.rb myappleid mypassword > devices.txt

Here’s the source of the script:

require 'rubygems'
require 'mechanize'

agent = Mechanize.new
page = agent.get("https://developer.apple.com/ios/manage/devices/index.action")

# Log in to Apple Developer portal if we're presented with a login form
form = page.form_with :name => 'appleConnectForm'
if form
  form.theAccountName = ARGV[0]
  form.theAccountPW = ARGV[1]
  form.submit
  page = agent.get("https://developer.apple.com/ios/manage/devices/index.action")
end

puts "deviceIdentifier\tdeviceName"

# Format each row as name,udid
rows = page.parser.xpath('//fieldset[@id="fs-0"]/table/tbody/tr')
rows.each do |row|
  name = row.at_xpath('td[@class="name"]/span/text()')
  udid = row.at_xpath('td[@class="id"]/text()')
  puts "#{udid}\t#{name}"
end

Click dl_devices.rb to download it. It simply tries to access the device management page, fills in a login form if required, and then extracts the device name and UDID using xpath expressions.

Troubleshooting

If you get an error like 'gem_original_require': no such file to load -- mechanize (LoadError), it means you don’t have the required ruby libraries installed. Install them with:

$ gem install mechanize

If the resulting data is missing or garbage, then it’s possible that Apple have changed the structure or flow of the developer portal. Try manually visiting https://developer.apple.com/ios/manage/devices/index.action and verify that you get asked to login, and then see your list of devices. If the structure of the page has changed, you could adjust the xpath references to match. If so, please send me the changes and I’ll update this page.

Solving AWS::S3::NoSuchBucket Error When Using Paperclip

2011-09-09T00:00:00+01:00

Very often when working on server side Ruby on Rails projects we need to support file uploads, such as pictures. To provide this support our gem of choice is Paperclip. We’ve used Paperclip for years and found it a huge improvement over attachment_fu which we used prior to it. However, in our last project it threw us a curve ball.

As we’ve done in dozens of previous projects we created an S3 bucket, configured Paperclip to use it and then started uploading files. However, uploads to the bucket failed with the following error appearing in the logs…

NameError (uninitialized constant AWS::S3::NoSuchBucket)

Petty strange since the bucket did exist.

So I used my trusty CyberDuck S3 client to connect to the bucket and upload some test images. I then tried to access these with the URL Paperclip was generating for them. For example http://s3.amazonaws.com/topsecretbucket/pictures/entity/3/pretty+picture+1.JPG?1311614390

The Amazon server returned an illuminating response…

<Error>
  <Code>PermanentRedirect</Code>
  <Message>
    The bucket you are attempting to access must be addressed using
    the specified endpoint. Please send all future requests to this endpoint.
  </Message>
  <RequestId>2EACG096167A1212</RequestId>
  <Bucket>topsecretbucket</Bucket>
  <HostId>
    dV/835QeLLt/g/t/Lhl+SCTVTGV3uo801DBUzw+tq85Nc+gTPDTDEiq3+oTwmKcM
  </HostId>
  <Endpoint>topsecretbucket.s3.amazonaws.com</Endpoint>
</Error>

Notice the endpoint Amazon recommend has the bucket name as a subdomain and not as part of the path.

Eureka!

The root of the problem was my bucket, which was configured to be located in the EU and not the US, as we’ve used in all our previous projects. Once we recreated a “US Standard” bucket the problem was solved. An examination of the Paperclip source revealed it only supports US based buckets.

As stated in the Amazon Simple Storage Service Developer Documentation…

Important

Amazon S3 supports virtual-hosted-style and path-style access in all Regions. The path-style syntax, however, requires that you use the region-specific endpoint when attempting to access a bucket. For example, if you have a bucket called mybucket that resides in the EU, you want to use path-style syntax, and the object is named puppy.jpg, the correct URI is http://s3-eu-west-1.amazonaws.com/mybucket/puppy.jpg. You will receive a “PermanentRedirect” error, an HTTP response code 301, and a message indicating what the correct URI is for your resource if you try to access a non US Standard bucket with path-style syntax using:

http://s3.amazonaws.com

A different Region endpoint than where the bucket resides, for example, http://s3-euwest-1.amazonaws.com and the bucket was created with the location constraint of Northern-California

The S3 module in Paperclip is hardcoded to use http://s3.amazonaws.com and so can only reference US based buckets.

Installing PostgreSQL on Lion

2011-08-09T00:00:00+01:00

OSX Lion ships with the PostgreSQL client tools, as well as a _postgres daemon user. I wanted to add a full PostgreSQL server in a way that was compatible with these tools and without creating an extra user account.

This recipe uses HomeBrew to do the heavy lifting. If you haven’t already got it, head over to http://mxcl.github.com/homebrew and install it.

First, make sure brew is up to date:

$ brew update

Start the install process:

$ brew install postgresql

This will take a few minutes to download and eventually end with something like this:

==> Summary
/usr/local/Cellar/postgresql/9.0.4: 2577 files, 34M, built in 77 seconds

Next, initialize the database storage with this command:

$ initdb /usr/local/var/postgres

The built-in psql command (and many apps) expect to connect to a local PostgreSQL database using a unix domain socket in /var/pgsql_socket. This is currently owned by the _postgres user so we need to fix that:

$ sudo chown $USER /var/pgsql_socket/
Password:

Now create a launchctl configuration to start the database automatically. First, copy the default config file:

$ mkdir -p ~/Library/LaunchAgents
$ cp /usr/local/Cellar/postgresql/9.0.4/org.postgresql.postgres.plist ~/Library/LaunchAgents/

Now, edit ~/Library/LaunchAgents/org.postgresql.postgres.plist in a text editor and look for this line:

<string>/usr/local/var/postgres/server.log</string>

Paste in the following lines immediately below it:

<string>-c</string>
<string>unix_socket_directory=/var/pgsql_socket</string>
<string>-c</string>
<string>unix_socket_group=_postgres</string>
<string>-c</string>
<string>unix_socket_permissions=0770</string>

Start the database server like this:

$ launchctl load -w ~/Library/LaunchAgents/org.postgresql.postgres.plist

Add the _postgres account as a database user (with no password):

$ createuser -a -d _postgres

Finally, verify that it’s all working by connecting:

$ psql template1 -U _postgres
psql (9.0.4)
Type "help" for help.

template1=#

You should also be able to connect with a TCP/IP connection to localhost, port 5432.

Creating Alpha Transparent Favicons on a Mac

2010-07-08T00:00:00+01:00

Normally I use a very old Mac program called IcoMaker to create favicons. The problem with IcoMaker is it can’t generate alpha transparent ico files. I recently had to design a favicon for StatusHub which was round, so the standard gif level of transparency wasn’t going to cut it. Time to roll up my sleeves and find some new, hopefully free, software…

Given that favicons are based on the Windows ico format there aren’t a huge number of options available to create them on the humble Mac. Most programs claiming to support the ico format don’t support alpha transparency. For completeness, here are the set of options which don’t work:

Fireworks CS4 (my standard tool of choice): it doesn’t support ico files at all.
Pixen: can read/write ico files but can’t save them with alpha transparency.
Pixelmator: can open but not save ico files.
png2ico: unix command line tool which unfortunately doesn’t compile on Mac OS X.

I also tried several online tools like Dynamic Drives Generator and the favicon.cc generator but they all had problems.

Eventually Gimp came to the rescue. Normally I steer clear of Gimp since I don’t like the interface but in this area it excelled. If you already have your alpha transparent png file all you need to is open it in Gimp then:

Select File > Save As…
Click on “Select File Type (By Extension)”
And select “Microsoft Windows Icon” – ico, its about half way down the list

If you are looking for more complete instructions on building a multi-resolution favicon, from scratch in Gimp, then check David Lane’s article.

When testing your favicons in Firefox don’t forget to clear the cache to see your new designs. Note, Firefox is also the best browser to confirm your ico file’s alpha transparency is working correctly. Safari displays them on a white background while Firefox uses a faded grey background.

HTML Editor Control of Choice

2010-05-26T00:00:00+01:00

Surprisingly there are very few decent, open source HTML WYSIWYG editor controls. There are a whole collection of them but most are out of date or way too feature rich and heavy.

I recently required a HTML WYSIWYG editor control for a project I was working on and so spent a couple of hours reviewing and testing the available options. Here are the few which are actually worth considering.

1) tinymce, version 3.2.4.1 (website)

“Tiny"mce weights in at a hefty 176K compressed. It is one of the most popular HTML editor controls around and is quite flexible with plenty of themes and plugins. It also quite active with a new release out every 2-4 months. It was easy enough to get up and running but I found the plug-ins fairly limited and styling it was just too much work.

2) FCKEditor, version 2.6.4 (website)

The file size of FCKEditor depends on the browser used to render it. In my tests it came in at about 72K compressed. This is one of the granddaddy HTML editors. It has lots of features and a large web following. Unfortunately I found it just too heavy to work with. The docs weren’t great and the download, all 1.5MB of it, was so cluttered with features, plugins and skins that it was too difficult to just get something simple going. It has gathered too many features in its life to suit my needs. The team are hard at work on a total rewrite (version 3.0) which may address this.

3) NicEdit, version 0.9 r23 (website)

A nice fairly light framework. Depending on which features you want to use it weights in between 25 – 32K. It uses the famfam icon set for its tool bar buttons (I’m a huge fan of famfam) and so has a nice clean look. As with the FCKEditor and tinymce editors, NicEdit has problems generating semantically clean XHTML. There is an experimental nicXHTML plugin which addresses this. It has known problems but so far it seems to be working fine for my needs.

It has some lovely behavior built right in such as auto-expanding the text area to ensure all the content is displayed. No more grippies! Overall I thought NicEdit was well put together.

4) wysihat, version 0.2 (website)

Built on Prototype this library weighs in at 53K (before compression). At the time of writing wysihat is at version 0.2. I had previously worked with 0.1 and the new version does fix a lot of problems. However, it still isn’t ready for prime time. On Firefox and Safari it works well but IE support lets it down. Having said this wysihat is, in my opinion, the future. It is small, lite and easy to extend. A project I’m rooting for and definitely one to keep an eye on. And the winner is…

In the end I went with NicEdit. Its just so much better than the existing heavy weights and wysihat needs some more love before its ready for production. I’m sure there will be some problems with NicEdit but I feel I’m starting from a good base. I’ll keep you updated on how NicEdit holds up as development continues.

Installing Phusion Passenger on Mac OSX

2010-04-09T00:00:00+01:00

Actually this is incredibly easy, assuming you have the gem manager installed (see here if you want instructions).

The phusion installer is excellent. All you need do is open a terminal and enter…

1 sudo gem install passenger
2 sudo passenger-install-apache2-module

Then just follow the on-screen instructions with the following caveats…

Caveat 1: Adding Phusion to Apache

The install suggests you add the following lines to you apache config (which can be found in /etc/apache2/httpd.conf )

1 LoadModule passenger_module /opt/local/lib/ruby/gems/1.8/gems/passenger-2.1.3/ext/apache2/mod_passenger.so
2 PassengerRoot /opt/local/lib/ruby/gems/1.8/gems/passenger-2.1.3
3 PassengerRuby /opt/local/bin/ruby

However, I suggest you add these to a new file in a separate conf file. Create a file with the following name and apache will load it automatically on startup.

1 /private/etc/apache2/other/phusion.conf

Tip: Restarting Apache

The phusion install asks that you restart Apache. This can be done using:

1 sudo apachectl -k restart

Caveat 2: Deploying your Application

Again the install program is very clear. It says you should add the following to your Apache config

1 <VirtualHost *:80>
2   ServerName www.yourhost.com
3   DocumentRoot /somewhere/public    # Be sure to point to 'public'!
4 </VirtualHost>

But remember to remove the comment “# Be sure to point to ‘public’!” or Apache will display the following error on restarted:

DocumentRoot takes one argument, Root directory of the document tree

Also, you should add the following after the virtualhost definition.

1 <Directory "/somewhere/public">
2   Options ExecCGI FollowSymLinks
3   AllowOverride all
4   Allow from all
5 </Directory>

This gives Apache access to your Rail app’s directory, getting rid of any annoying 403 ‘access denied’ errors.

That’s pretty much it. Remember, by default phusion will run your application in production mode. So ensure your database.yml file has a production database configured.

Forcing IE to Redraw List Items

2010-01-07T00:00:00+00:00

Javascript is essential in allowing developers to tailor an intuitive and accessible experience for users. Web sites are becoming a lot more dynamic and javascript is key to this progression (side note: I’m not a fan of Flash, it makes my Mac work too hard). However, as usual developers encounter lots of browser interop issues – most of them with Internet Explorer.

Recently I was using javascript to dynamically add extra data to an unordered list item. This worked fine in FireFox and Safari. However, it turns out Internet Explorer 6 & 7 are very intolerant of content being added, or removed, from list elements. The problem is they don’t recognize the list item has resized and therefore don’t redraw it.

Luckily it is possible to give IE a nudge, so that it redraws the list. To do this you need to dynamically add a new child element to the list… and then remove it again. Here is an example javascript function, based on Prototype, which does exactly this. This function is now part of my toolbox and I call it after I add extra content to a list item.

 1 /* IE 6 & 7 are very intolerant of extra content being
 2 * added/removed to elements so a redraw needs to be
 3 * forced.
 4 */
 5 function forceListRedraw(listItem) {
 6   if (Prototype.Browser.IE) {
 7     try {
 8       var n = document.createTextNode(' ');
 9 
10       listItem.appendChild(n);
11       listItem.removeChild(n);
12 
13       list = listItem.up('ul');
14       list.appendChild(n);
15       list.removeChild(n);
16     } catch(e) { }
17   }
18 }

Migration Helpers

2009-09-30T00:00:00+01:00

Working on a migration recently I noticed the same set of operations being done repeatedly. My first reaction was to DRY up the code by creating a utility method, within the migration. But I was sure I’d end up needing this method in future migrations. But the question was, where should I put it?

The answer was to create a migration helper!

First, I created lib/migration_help.rb containing the following:

1 module MigrationHelper
2   def do_good_stuff()
3     say "boy, this method will be useful..."
4   end
5  
6   def undo_good_stuff()
7     say "actually on second thoughts it was stupid..."
8   end
9 end

Then in my migration I added ‘require’ and ‘extend’ to import the MigrationHelper module.

require “migration_helper”

 1 class AddNewFeatureColumns < ActiveRecord::Migration
 2   extend MigrationHelper
 3  
 4   def self.up
 5     do_good_stuff
 6   end
 7  
 8   def self.down
 9     undo_good_stuff
10   end
11  
12 end

And that was it! The helper method names have been changed, to protect the innocent, but you get the idea. Not exactly rocket science but usefully if you need to condense your migrations.

For more information I suggest you check out the excellent discussion on migrations, in the RailsGuides; after reading it I changed all my migration helper ‘puts’ statements to ‘say’ statements. It nice to get those little details right.

Git Stash for Quick Patches

2009-03-26T00:00:00+00:00

For source control management I’m currently using git, actually my repository is hosted on GitHub (a fantastic service and highly recommended). Git can take a little getting used to but it has several great features that subversion just doesn’t have. My current favorite is the ability to ’stash’ work-in-progress.

We’ve all been there; you’re happily working on the code, halfway through a big change, when someone asks you to quickly fix a bug. Traditionally your only option was to checkout the repository head into another directory so you could make your change there – without being forced to commit or revert the current changes you were working on. But no longer, ‘git stash’ to the rescue!

Now, you can quickly park the work you are doing using…

git stash

This will record the current state of the working directory, saving your modifications and reverting the working directory so that it matches the head of the repository. Now you can make the small tweak to fix the bug without losing your changes. The ‘git commit’ and ‘git push’ the fix as normal. When you are happy the patch is complete type…

git stash apply

Now your saved modifications are restored and you can continue adding that great feature you were working on.

Of course if losing your changes wasn’t a concern you could have used…

git reset --hard HEAD

And all your local changes would have been lost, reverting your working copy to the latest repository head.

Useful Git Resources

Authur Koziel has some great git configuration hints. I love the bash shortcuts, color coding and textmate integration.
GitHub have a handy git cheatsheet and if you want to dig deeper there’s alway the git user’s manual.

TextMate Trimming & Shortcuts

2009-03-17T00:00:00+00:00

TextMate is my development environment of choice these days. It supports the concept of bundles as an extension mechanism. Several great bundles are shipped with TextMate and I find they cover most of my needs.

Additional ones can be installed and there are several ones which aid Rails development. For a while I ran RubyAMP but I found TextMate’s memory usage went through the roof. So currently I just run the vanilla set.

Hiding Unwanted Bundles

I stripped down the bundles which are available in TextMate. It ships with a wide range of bundles which just get in the way when you just want to work with HTML, CSS and Ruby/Rails.

If you want to do the same go to the bundle editor, via the ‘Show Bundles’ option in the ‘Bundles’ menu and press the ‘filter’ button. For full details refer to the bundles manual.

Useful Keyboard Shortcuts

Key Modifer Legend :⌘ = Command, ⌥ = Option, ^ = Ctrl, ⇧ = Shift, ⎋ = Esc, ␣ = Space, ↩ = Return/Enter, ⇥ = Tab

Global Shortcuts (work on most types of files)

^h = Brings up help on the current word. For example W3C reference help for HTML or Rails Doc for rb files.
⌥ ⎋ = Auto complete. Suggests possible attributes, etc. For example type <img ⌥ ⎋ and a popup with possible attributes appears.
⇥ = Expand snippet. For example in an ruby file, type : and then press tab. This expands to :label => value.

HTML Specific Shortcuts

⌘& = Pop-up entity/escaping commands menu.
⌥⌘. = Insert close tag.
⌃< = Make closing/opening tags for word to left of cursor.
^␣ = Non breaking space.