Tomi Hiltunen

Enthusiastic app developer in Google Go & HTML5

maanantai 18. maaliskuuta 2013

Heads up with GAE blobstore!

The problem

I recently ran into problems with Google App Engine blobstore. My HTML5 page had a multipart form for creating content on my app. The form included various inputs for name, coordinates and also a file input for a photo. The form was posted to the blobstore generated upload URL... and it crashed.

After experimenting a bit with my code I narrowed down the cause to be with blobstore's method ParseUploads(). The problem exists only when at least one of the text inputs contained a value with non-ascii characters such as the scandinavian "ö".

I ran a search on Google App Engine's issue tracker on "blobstore go" and found out that it is an actual reported bug in the environment.

The same problem does not exist when there is no non-ascii characters in the inputs. The problem does not exist either with regular form uploads (without file inputs) read with r.FormValue("...").

The solution (for now)

I went around the problem by separating the photo uploads from the other data. My form now does not include a file input. I replaced that with a button connected to the Plupload JavaScript plugin. The upload is handled by a separate handler and the form posts only the text inputs to another handler.

Now my content can have the photos added and names/descriptions can contain non-ascii characters. This satisfies my need for now!

Using URL slugs

What are slugs in URLs?

Slugs are a part of URLs that make the URLs more human-readable and search engine friendly. A URL with a slug gives the viewer a better idea of what is behind the link before even clicking on it.

Lets say that you have an article named "Tower of London". The unique URL for this article without the slug could be:

http://example.com/articles/21

Now with a slug generated from the title of the article:

http://example.com/articles/21/tower_of_london

Seems a lot more informative? Do you find the topic more obvious in the last example?

Essentials in generating a slug

In this article I will discuss just the idea of generating a slug from the title of your content. It is also suggested that other important keywords should be included in the slug for further SEO friendliness. I'm happy using just the title without extra keywords, but feel free to include them in your own app.

The key is that slugs should consist only of characters that do not need to be URL encoded.

There's different variations for generating slugs. I find it more visually pleasing to replace all white-space characters with underscores. Some replace other punctuations with a dash or underscore but I rather leave them out of the slug all together.

How I do it

Here's an algorithm used by me for slugging the titles:
  • Check for all non URL-friendly characters with reg-exp [^A-Za-z0-9\s-_]
    • Non-friendly character was found!
    • See if it can be transliterated
      • Yes?
        • Replace with the latin counter part
      • No?
        • Replace with an empty string
  • Trim all the leading and trailing white-space characters
  • Replace all 1...* characters long white-spaces with an underscore. \s+
  • Convert the string to lower-case
  • Done!

In Go

import (
    "strings"
    "regexp"
)

/*
 *  Dictionary of accented characters and their transliterations.
 *  Don't consider this dictionary complete!
 */
var dictionary = map[string]string {
    "Š": "S",
    "š": "s",
    "Đ": "Dj",
    "đ": "dj",
    "Ž": "Z",
    "ž": "z",
    "Č": "C",
    "č": "c",
    "Ć": "C",
    "ć": "c",
    "À": "A",
    "Á": "A",
    "Â": "A",
    "Ã": "A",
    "Ä": "A",
    "Å": "A",
    "Æ": "A",
    "Ç": "C",
    "È": "E",
    "É": "E",
    "Ê": "E",
    "Ë": "E",
    "Ì": "I",
    "Í": "I",
    "Î": "I",
    "Ï": "I",
    "Ñ": "N",
    "Ò": "O",
    "Ó": "O",
    "Ô": "O",
    "Õ": "O",
    "Ö": "O",
    "Ø": "O",
    "Ù": "U",
    "Ú": "U",
    "Û": "U",
    "Ü": "U",
    "Ý": "Y",
    "Þ": "B",
    "ß": "Ss",
    "à": "a",
    "á": "a",
    "â": "a",
    "ã": "a",
    "ä": "a",
    "å": "a",
    "æ": "a",
    "ç": "c",
    "è": "e",
    "é": "e",
    "ê": "e",
    "ë": "e",
    "ì": "i",
    "í": "i",
    "î": "i",
    "ï": "i",
    "ð": "o",
    "ñ": "n",
    "ò": "o",
    "ó": "o",
    "ô": "o",
    "õ": "o",
    "ö": "o",
    "ø": "o",
    "ù": "u",
    "ú": "u",
    "û": "u",
    "ý": "y",
    "þ": "b",
    "ÿ": "y",
    "Ŕ": "R",
    "ŕ": "r",
}

/*
 * Creates a lower-case trimmed string with underscores for white-spaces.
 *
 *      - Converts to lower case.
 *      - Trims the leading/trailing white spaces.
 *      - Converts applicable accented caharcters to non-accented and removes invalid ones.
 *      - Converts leftover white-spaces to underscores regardles of type.
 */
func Slug(original string) (edited string) {
    // Remove invalid characters
    re, _ := regexp.Compile(`[^A-Za-z0-9\s-_]`)
    edited = re.ReplaceAllStringFunc(original, convertAccent)
    // Trim leading and trailing white-space
    edited = strings.TrimSpace(edited)
    // Convert all white-spaces to underscores
    re, _ = regexp.Compile(`\s+`)
    edited = re.ReplaceAllString(edited, "_")
    // All done!
    return strings.ToLower(edited)
}

/*
 * Converts accented characters if found from the dictionary.
 * Otherwise will replace the character with an empty string.
 */
func convertAccent(found string) (string) {
    if newValue, ok := dictionary[found]; ok {
        return newValue
    }
    return ""
}

Optimizing images on GAE blobstore using Go

So your app allows users to post image files to the App Engine blobstore?

About App Engine blobstore

Blobstore is a service provided by Google App Engine for storing files or "blobs". The blobstore provides an easy method for allowing users upload files to your app. On this article I will focus solely on image files and how to optimize them after they are uploaded to the blobstore.

Reasons for optimizing the images

Optimizing the images may mean scaling down the image dimensions and changing the compression rate. This affects the image's file size. Smaller file size means lower storage costs and shorter download times. Shorter download times make happy users.

How to do it

Please note: I expect that you already know the basics of using blobstore for uploading files!

Step 1) Download my image optimizer package from GitHub

Step 2) Import the package to your project

import "github.com/tomihiltunen/gae-go-image-optimizer"

Step 3) Where you would normally call "blobstore.ParseUploads()"

// Create settings for the optimization.
optimizerOptions := optimg.NewCompressionOptions(r) // r *http.Request

// Set maximum image dimension
// 0 = no change (default)
optimizerOptions.Size = 1600

// Set image quality
// 100 = no compression (defaults to 75)
optimizerOptions.Quality = 75

// Call the ParseBlobs method.
// Return values are similar to blobstore.ParseUploads().
blobs, other, err := optimg.ParseBlobs(optimizerOptions)

// Do the rest
...

Considerations

Reading and writing to the blobstore is a rather lengthy operation. Optimizing images immediately during upload process will make your app appear slow to users. In this case the image would be first uploaded to the blobstore, the request redirected to your app's handler, image read from the blobstore, optimized and finally, the new image is written to the blobstore and the original one deleted. As you know, responsiveness of your app affects directly the experience of using your app.

One option for eliminating the slow response times is to have a task queue for the optimization process. When the blob is uploaded your handler will just add the object's and blob's keys to the task queue. When the queue has finished the optimization it will simply replace the blobkey to the objects properties. The old blob can be used as the image until the task queue has finished the optimization.

Optimizing on the client-side

If you do not need server-side optimization, you can use plugin like Plupload which has the options for optimizing image dimensions and compression rate before uploading the image to the server. Benefits of this approach is that the upload time is shorter and you don't have to use your server resources for the optimization process as you can make the client do the heavy lifting.

The rebirth of my blog

I finally got around blogging. I created this blog way back in Feb 2012 but haven't since published anything. I had completely forgotten my own blog. I felt that now it's a high time for me to give a new life to my blog.

I will be blogging on my findings and experiences as an enthusiastic app developer and being a startup entrepreneur. Mostly my development related posts will be about my passions: Google Go, HTML5, CSS3 and (OO) JavaScript. I use those techniques on a daily basis as I develop my own apps.

A little about my background

I've been developing apps for roughly eight years now. While developing my own apps, I have worked for small and multi-national organisations on various projects ranging from core-level mobile Linux platform testing to UI development & design.

With one of my ideas I went to the finals of three startup competitions in the early 2012:
  • LeWeb London (top-16 / 600 applicants)
  • GMIC G-startup in Beijing (top-12 / 200 applicants)
  • Kasvu Open where my idea shared the #2 place
I was also with the winning team in the X-MAS hackathon organized in cooperation by Nokia & Microsoft in July 2012.

For about 7-8 years I was developing mainly with PHP. Recently I traded PHP completely for Go, a type-safe compilable language by Google. Ever since I started using Go I was hooked. Now all my apps use Google App Engine + Go combination.

That all said...

I hope you enjoy reading my blog. I'm looking forward for some interesting discussions in the comments!