Creating Python dictionaries with “reduce”

In the last few installments (first, second, third, and fourth) of this series, we saw how “reduce” can be used to build up a scalar value (such as a number or string), or even a simple collection, such as a list (in Python) or an array (in Ruby).  The jewel in the data-structure crown for these high-level languages is known by many names: Dictionary, hash, hash table, hash map, and mapping.  I tend to use these quite a bit, and I know that I’m not alone; hashes are easy to use and work with, have O(1) lookup characteristics, guarantee the uniqueness of their keys, and make the code self-documenting.  What’s not to like?

There are times when you might want to build a dictionary step by step, and “reduce” can help you to do that.  I should note that recent versions of Python offer “dictionary comprehensions,” which are one of my favorite features of the language, and can be used similarly to “reduce” — and probably with less confusion among new programmers.  Nevertheless, I find it interesting and instructive to see how we can use “reduce” to create a data structure that we wouldn’t normally associate with this function.

I should note that some of the things you’re going to see here are not really recommended coding practices, particularly in Python.  But they will be fun, which is important, no?

Let’s start with Python: If I want to create a dictionary using “reduce”, I have a few different options.  Perhaps the first and easiest one is not exactly what you might have had in mind, namely passing the “dict” function (which creates a new instance of “dict” — i.e., a new dictionary) a list of two-element tuples.  For example, I can say:

dict([('a', 1), ('b', 2)])

and I get:

{'a': 1, 'b': 2}

So if I can tell “reduce” to emit a list of tuples, I can then pass that list of tuples to “dict”, and get a dictionary back.  Let’s try that:

reduce(lambda output, current: output + [(current, ord(current))], 'abc', [])

In the above case (and in all of the examples we’ll use here), I’m trying to build an oh-so-useful dictionary in which the keys are the letters a, b, and c, and the values are the ASCII codes for those letters.  In the above Python code, I iterate over the string ‘abc’, which is a three-element sequence of those three letters.  For each letter, I return the current value of “output” (which is guaranteed to be a list), plus a new, single-element list.  That single-element list is a tuple consisting of the current letter and its ASCII code.  So the output of the above “reduce” call will be:

[('a', 97), ('b', 98), ('c', 99)]

which, when we feed it into dict():

>>> dict([('a', 97), ('b', 98), ('c', 99)])
{'a': 97, 'b': 98, 'c': 99}

Voila!  We’ve created a dictionary.  (By the way, the “dict.items” method, which is often used to iterate over the keys and values of a dictionary, returns a list of tuples in precisely this format.)

So this is nice, but perhaps there’s a way to have “reduce” build up a dictionary all by itself?  The answer is “yes, but” — because Python tries hard to stop us from modifying data structures within a lambda.  But if we’re willing to be a bit weird, we can get around that.  How? By taking the dictionary that we received from the previous iteration (i.e., “output”), turning it into a list of tuples with dict.items, adding that list to our current pair, and then turning the whole thing back into a dictionary.

reduce(lambda output, current: dict(output.items() + 
                     [(current, ord(current))]), 'abc', {})

Note that while this is unwieldy, it doesn’t violate one of the key tenets of functional programming, namely treating data as immutable.  However, putting this sort of code in a production system will lead to hatred from your colleagues, job security, or (if you’re really lucky) both.  Oh, and it’s probably rather inefficient as well, although I’m not crazy enough to actually benchmark it.

That said, consider what we’ve done here: We create a new dictionary with each iteration, passing it to “current”.  The new dictionary is created by taking the old one, breaking it into a list of tuples, adding to it a new tuple, and then turning it all into a dictionary again.

There is at least one other way to do this, but it’s going to make the above code seem like beautiful, classic Python in comparison: Remember that we cannot assign to a dictionary within a lambda.  However, we can invoke methods, including the dict.update method, which lets us merge one dictionary into another.  The thing is. dict.update, like most methods in Python that modify data structures, returns None.  If we’re willing to take some risks (and why stop now), we can use the following code to modify our existing dictionary and then return it to the next iteration:

reduce(lambda d, current: d.update({current : ord(current)}) or d, 'abc', {})

The above code takes advantage of the fact that “and” and “or” are short-circuit operators.  Thus, we first invoke d.update on our dictionary.  That returns None.  Our “or” operator then says, “Well, I’d better go to the second argument, because the first one returned a false-y value.  Sure enough, our dictionary — because it isn’t empty — is a true value, which is what the statement returns.  And sure enough, we get our dictionary.

A final way to do this would be to hide the assignment of the dictionary behind an external function.  That is, we have a function do the dirty work for us.  That’ll certainly work, although I see it as somewhat less fun than lambdas and trying to work around their limitations.

def update_and_return(d, new_key, new_value):
    d.update({new_key: new_value})
    return d

>>> reduce(lambda output, current: update_and_return(output, current, ord(current)), 'abc', {})

{'a': 97, 'b': 98, 'c': 99}

Fun stuff!  And, perhaps, the worst method ever for getting an ASCII table into a dictionary.

Next time, we’ll see how to do this with Ruby.

Creating collections with “reduce”

In the first few parts of this series (first, second, and third), I introduced the “reduce” function, and showed how it can be used in a number of ways. However, in all of the examples we have seen so far, the output from our invocations of “reduce” were integers or strings. If we reduce with a “sum” function, then we get an integer representing the sum of all integers.  If we reduce over a measurement, such as the size of an integer or the length of a string, we can implement “min” or “max”.

Another interesting and clever thing that we can do with “reduce” is to create an array (in Ruby) or list (in Python).  Think of it this way: The output of each iteration of “reduce”, which is then passed to the next iteration, can be any data structure we wish. If the output from the iteration is an array, then that’s what will be passed to the next iteration.

In other words, we can use “reduce” to build up an array, with each iteration emitting that same array, perhaps with additional elements added to it.  For example, I can create a new array whose contents are identical to the original one on which “reduce” was invoked:

['a', 'b', 'c'].reduce([]) {|output, current| output << current }

By adding each element to the beginning, I can reverse the array:

['a', 'b', 'c'].reduce([]) {|output, current| output.unshift(current) }

I can create a nested array, too:

['a', 'b', 'c'].reduce([]) {|output, current| output << [current, current] }

“reduce” is thus a great way to create an array, building it up piece by piece.  True, we need to pass “reduce” an empty array as a parameter, and then each iteration needs to return the current value of the array.  But other than that, we can add whatever we want to the array, and modify it in whatever ways we want.  If I want to do something more complex than the above, or if I want to be able to debug individual lines of my block, I can even use do-end, thus spreading the block across multiple lines.  (And yes, I know about the late Jim Weirich’s great suggestion, to use curly braces for returning values.  But if you’re going to have multiple lines in your block, then do-end still rings truer for me.)

Let’s see how we can do the same thing in Python.  First, we can return a list:

reduce(lambda output, current: output + [current], 'abc', [])

We start (using the optional final argument) with an empty list.  Then, in each iteration, we take the current element, turn it into a one-element list, and then append it to the existing list.  Note that we cannot use list.append to add the current element, because list.append returns None, like many other methods that change mutable objects in Python.

Using “reduce” to reverse the list is just as easy, by reversing the order in which we add things to the list:

reduce(lambda output, current: [current] + output, 'abc', [])

Finally, if we want to return a list of lists, we can do that fairly easily, just add a list of lists:

reduce(lambda output, current: output + [[current, current]], 'abc', [])

As you can see, “reduce” makes it possible for us to, in a single line, create a collection based on another set of inputs.  The output can contain any types that we want; in these simple examples, we created lists of strings.  But we could create lists of dictionaries, or of more complex objects, just as easily.

In the next installment, we’ll see how to create hashes/dictionaries using reduce.



Implementing “min” and “max” with “reduce”

This is the third installment of my “reduce” series of blog posts.  For the first, see here, and for the second, see here.

If you have been reading this series, then you know that “reduce” can be used to sum numbers, or to calculate scores.  In that sense, “reduce” justifies its name; we’re invoking a function repeatedly on each element of a collection, boiling the collection down to its essense, as defined by our function.

But “reduce” can also be used to to other, more interesting things.  One of them, mentioned in the Ruby documentation, isn’t an obvious candidate for beginners with “reduce”, namely implementing “min” and “max”.

In a world without functional programming or reduce, we could implement a “min” function as follows in Ruby:

def mymin(items)
  output = items[0]           # Assume the first item is the smallest
  items[1..-1].each do |item| # Iterate through all remaining items
    if item < output
      output = item           # This item is smaller, so keep it
  output                      # So far, this is the smallest we've seen

The definition, if you’re somewhat new to Ruby, works as follows: The “min” function takes an enumerable sequence of items. We assume that the first item is the smallest one, simply to make life easier for ourselves.  We then iterate over the remaining items, checking to see if the current one is smaller than the current winner.  Each iteration thus leaves “output” with its existing value, or replaces it with something smaller.

Now, the above code works just fine — but it’s a bit long and inelegant just to find the minimum value in an enumerable.  After all, it should be possible to implement the same algorithm we we used above in the implementation of “mymin”, but somehow more elegantly and concisely.

That’s where “reduce” comes in.  We saw in an earlier blog post that when we use “reduce” to sum the numbers in a collection, is sort of like saying ((((1+2)+3)+4)+5).  Well, let’s say that we had a function “smaller” that would return the smaller of two numbers.  Then, we could use “reduce” in the following way:


Oh, is that hard to read?  Yeah, I thought so.  So instead, we can do the following:

items.reduce {|smallest, current| smaller(smallest, current) }

Each time “reduce” invokes our block, it keeps the smaller of its two parameters.  The first parameter, which I’ve here called “smallest”, contains the smallest value that we’ve seen so far. The second parameter, “current”, contains the current value from the collection.

That one line certainly looks nicer than our original implementation of “mymin”, I think. There’s just one problem: We haven’t defined any “smaller” method!  Fortunately, we can use Ruby’s ternary operator to have a tiny, one-line if-then-else statement that does the same thing.  In other words:

items.reduce {|smallest, current| smallest < current ? smallest : current }

The moment you realize that “reduce” can be used to hold onto only one of the previous values, you begin to see the opportunities and possibilities that it offers. For example, we can find the shortest or longest word in an array of words:

words = 'This is a sample sentence'.split
words.reduce {|shortest, current| (shortest.size < current.size) ? shortest : current }
=> "a"

words.reduce {|longest, current| (longest.size > current.size) ? longest : current }
=> "sentence"

Now, can we do the same thing in Python?  The builtin “reduce” function works similarly to Ruby’s Enumerate#reduce, as we’ve seen in previous posts I wrote on the subject.  However, the function that we pass to Python’s “reduce” is more limited than a Ruby block, especially if we use a lambda (i.e., anonymous function).

However, we do have a trick up our sleeve: Python provides an analogous version of Ruby’s ternary operator, which is meant for precisely these occasions.  I generally tell people to avoid this operator completely, because it causes more problems and confusion than necessary, and probably means that you’re doing something that you shouldn’t be in Python.  It works as follows:

>>> x = 'a'
>>> is_an_a = "yes" if x == 'a' else "no"

>>> is_an_a

The oddest thing for me about this form of “if” in Python is that the condition is in the middle of the line.  It’s not the postfix “if” of Perl and Ruby, and it’s not the prefix “if” that everyone is used to, but something else entirely.  And as such, it’s usually a bad idea to have in your code.  But right now, it fits perfectly.

Here, then, is a version of “min” in Python, using “reduce”:

>>> items = [5,-5,10,1,100,-20,30]
>>> reduce(lambda smallest, current: smallest if (smallest < current) else current, items)

And sure enough, we get the lowest value!  Next time, we’ll produce even more complex outputs, with more complex data structures.

Calculating Scrabble scores with “reduce”

This is the second installment of my series of blog posts on the “reduce” function/method.  For an introduction, see here.

I love to play Scrabble — or more commonly nowadays, I play Words with Friends on my phone. (I often say that the game should instead be called, “Words with people who used to be your friends,” given the competition that can result.)  The game, if you’re not familiar with it, is simple enough to understand: You choose seven English letters at a time from a bag or pool of letters.  Each letter has a different point value.  Your aim is to create, with your seven letters, the highest-scoring word that you can in each turn.

Fine, it’s a bit more complex than that: Your word needs to hook onto an existing word or letter, and you can get bonus points by completing more than one word, and there are squares which multiply your score, by doubling or tripling the value of an individual letter or the entire word.  I’m going to ignore all of these rules for now.

Let’s say that I want to write a Ruby program that will calculate the score of a given word in Scrabble. How can I do that?

Before I can begin, I’m going to need to need a list of the points that you get for each letter.  Because we’re dealing with a mapping between two values — a letter and its points — the easiest solution would be a hash:

points = {'a'=> 1, 'b'=> 3, 'c'=> 3, 'd'=> 2, 'e'=> 1, 'f'=> 4, 'g'=> 2, 'h'=>
 4, 'i'=> 1, 'j'=> 8, 'k'=> 5, 'l'=> 1, 'm'=> 3, 'n'=> 1, 'o'=> 1, 'p'=> 3,
 'q'=> 10, 'r'=> 1, 's'=> 1, 't'=> 1, 'u'=> 1, 'v'=> 4, 'w'=> 4, 'x'=> 8, 'y'=>
 4, 'z'=> 10}

Once we’ve defined our “points” hash, we can start calculating the number of points that we’ll get for a given word.  If you aren’t familiar with “reduce”, but have been programming in Ruby for at least a short while, you’ll probably realize that this is easily accomplished with a loop.  That is, we can iterate over each letter in the string, getting the point value for that letter, and adding it to the total.  For example, we could do something like this:

word = 'encyclopedia'
total = 0 
word.each_char do |letter|
  total += points[letter]

After the above loop, we’ll find that the total score is 22 points.  That’s fine, and it’ll work… but by using “reduce”, we can do the same thing in a single line:

word.split('').reduce(0) {|total, current| total + points[current]}

Now, how does the above work? First, it takes the string “word”, and turns it into an array, in which each letter is an element.  Then, we use “reduce” to calculate a score: Since it’s a numeric score, we’ll initialize the total with 0.  Then, for each letter in our word, we add the number of points associated with that word to our total.

To do the same thing in Python will require surprisingly similar code. First, we create a dictionary:

points = {'a': 1, 'b': 3, 'c': 3, 'd': 2, 'e': 1, 'f': 4, 'g': 2, 'h':
 4, 'i': 1, 'j': 8, 'k': 5, 'l': 1, 'm': 3, 'n': 1, 'o': 1, 'p': 3,
 'q': 10, 'r': 1, 's': 1, 't': 1, 'u': 1, 'v': 4, 'w': 4, 'x': 8, 'y':
 4, 'z': 10}

Then we run reduce in a similar way. Because strings in Python are sequences, we don’t need to turn our word into an an array before iterating over it; we can invoke “reduce” directly on the string:

>>> word = 'encyclopedia'
>>> reduce(lambda total, current: total + points[current], word, 0)

This is a simple demonstration of where “reduce” can be useful. I’m looking to apply the same operation on the elements of a sequence of data, adding the latest value to the total.  When I get to the end, I’d like to see what has accumulated so far.

In the next installment, we’ll see how we can use “reduce” to implement some other functions that we wouldn’t normally associate with functional programming, let alone with something called “reduce”.


Understanding “reduce” (first in a series)

One of the notable things about MIT’s computer science curriculum, at least back when I was studying there, was that you didn’t learn any “practical” programming languages.  Our work was all done in either Scheme (a dialect of Lisp) or in CLU (an early object-oriented language).  I can’t say that I have too many memories, let alone fond ones, of CLU.  But I definitely drank the Kool-Aid about Lisp, and have long believed that it has always represented the pinnacle of programming languages.  Things that I learned long ago in Lisp are only now becoming standard in popular languages.

For example, functional programming has become increasingly popular in the last few years, for a variety of reasons including the shrinking effects of Moore’s Law — and the resulting need to have multiple, immutable copies of your data in the computer, distributed across multiple processors.  Lisp has long had many functional capabilities; it was Lisp that first introduced me to such functions as “map”, which I use multiple times each day.  However, my regular uses of “map” are rarely in Lisp; rather, they’re in Python and Ruby, the languages that I use most in my day-to-day work.

When I teach classes in Ruby and Python, I spend a fair amount of time talking about functional programming techniques.  It might sound funny to discuss functional programming in two languages that are so clearly object-oriented, but I actually find it quite natural.  Sure, I create classes and throw objects around.  But if I have an array of values, then it’s very fast and easy for me to process them with functional techniques.  Python’s list comprehensions have basically taken the place of the “map” and “filter” functions, so while those exist, they’re not as necessary any more.  But once you understand such functions as “map” and “filter”, you’re poised to do all sorts of amazing things.

Perhaps the most intriguing of the functions from this school is “reduce”.  I have found, consistently over time, that “reduce” is the function most likely to surprise and confuse newcomers to this type of programming.  That’s because it’s easy to confuse what “reduce” is doing, and to forget how flexible its output can be.

I’ve thus decided to write a series of blog posts about “reduce”, in both Python and Ruby.  I’ll start with the simple stuff, and then move ahead with increasingly complex tasks.  You won’t necessarily start to use “reduce” all of the time, but if you’re like me, you will find all sorts of interesting uses for it.  I tell people that I tend to use “reduce” about once every six months — but when I do use it, it really saves the day.

In Ruby, we can use the “reduce” method (also available as “inject”, to satisfy people from the Smalltalk world) on any enumerable object.  The invocation looks like this:

[1,2,3,4,5].reduce(0) {|a, b| a+b}

I’ll explain what this all means in a moment.  But the most important change that I can already make is to use better names for the block parameters:

[1,2,3,4,5].reduce(0) {|total, current| total+current}

Ruby goes through each element of the enumerable, invoking the block on each element. Described in this way, we might confuse “reduce” with “map”.  However, in “map”, the output is an array of the same length as the input.  By contrast, with “reduce”, the output of each iteration is remembered for the next time around.  That is, the value of “total” in each iteration is the block’s result from the last iteration.  

The initial value of “total”, in the first iteration, is the parameter value that we pass, which is 0 in this case.  If you don’t pass a parameter, then “total” is initialized with the first element of the enumerable, and the first iteration (i.e., the first application of the block) takes place on the second element.  This is fine in the above example, but depending on the output you want, failing to pass a parameter, or passing one of the wrong type, can make a big difference.

You can also think of “reduce” as a sort of “join”, but one that evaluates its inputs and operations.  So instead of getting the string “1+2+3+4+5”, you get the result of actually invoking 1+2, and then (1+2)+3, and then ((1+2)+3)+4, and then finally (((1+2)+3)+4)+5. This is why some Lisp versions call this “fold”, rather than “reduce”.  I still don’t quite get why Smalltalk people call it “inject”, and thus never use that term in Ruby (except when introducing the five rhyming functional methods — select, detect, collect, inject, and reject — because it’s so much fun to say). But the effect, once you internalize it, can be used in many interesting ways.

Python doesn’t have a “reduce” method on sequences, but does have a builtin “reduce” function that can be invoked on sequences.  (In Python 3, the “reduce” function was moved to the “functools” module — which is better than the fate Guido had originally planned for it.)  Python’s “reduce” is similar to the one in Ruby.  To sum numbers, we say:

>>> numbers = range(10)
>>> reduce(lambda total, current: total + current, numbers)

As you can see (I hope), the use of “lambda” to create an anonymous function is analogous to the use of a block in Ruby.  In Python’s “reduce” function, you first pass the function you wish to invoke on the sequence, and then the sequence itself.  If you wish to pass an initial value for “total”, you can do so with an optional third parameter:

>>> reduce(lambda total, current: total + current, numbers, 10)

The classic use of “reduce” is to sum integers, as we saw above.  But we can, of course, perform additional types of operations, and produce additional types of output.  And to be honest, that’s where “reduce” starts to get more intriguing.  Any operation that you want to perform on an enumerable, such as an array, set, or range, and apply cumulatively to its elements, makes for a good choice for “reduce”. By experimenting with the input enumerable, the initial value of “total” that we pass as a parameter, and the block, we can do many interesting things.

In coming posts, I’ll explore some more of these ideas, and give you a tour of “reduce” in both Ruby and Python that will hopefully open your eyes to some of them, and give you a sense of where “reduce” can help to improve your thinking, and your code.

Benchmarking old-style and new-style Python classes

It has been many years since Python developers were really supposed to worry about new-style vs. old-style classes.  There is only one style (new) in Python 3.x, and even in Python 2.x, old-style classes have not been recommended for many years.  Nevertheless, I mention old-style classes in my Python courses, mostly so that participants will understand the potentially serious implications of creating classes without inheriting from object.  For example:

>>> class Foo(object): pass
>>> type(Foo)

>>> f = Foo()
>>> type(f)

The above is the way that modern Python programmers define classes.  This is the preferred way, for sure; if you’re writing old-style classes, then you’re almost certainly doing something wrong.  But it’s so easy to create an old-style class in Python — all you have to do is forget to inherit from “object”:

>>> class Foo(): pass
>>> type(Foo)

>>> f = Foo()
>>> type(f)

As you can see, the fact that I created an old-style class directly affects the types of objects that I have created, and thus their capabilities.  For many years, it has been seen as a mistake to create old-style classes; not only are you missing out on new functionality, but you are creating objects that behave differently from the rest of objects in Python.

I was just teaching a Python class at a company that has a fair amount of legacy Python code.  It turns out that this legacy code includes a large number of old-style classes. The company asked me whether it was worth upgrading all of their old-style classes to use new-style classes; my answer was that (1) if it ain’t broke, don’t fix it, (2) it’s hard to know whether the upgrade would be trivially easy or impossibly hard, and (3) you’ll likely want to upgrade these classes over time, doing so incrementally.

Someone then asked me whether there is a performance difference between old-style and new-style classes, in order to evaluate the importance of doing such an upgrade project.  I had to admit that I wasn’t sure, and couldn’t find anything online (after doing a quick search) on the subject.  I thus decided to do a small benchmark to see what might be faster (or slower).  I’m not an expert in benchmarking, but I did want to check the basic speed of (1) object creation, (2) inheritance, and (3) implementation of __repr__.

The results surprised me: New-style classes are substantially faster.  Here is the benchmark that I ran on the new-style class:

class Person(object):
    def __init__(self, first_name, last_name):
        self.first_name = first_name
        self.last_name = last_name
    def fullname(self):
        return self.first_name + " " + self.last_name
    def __repr__(self):
        return self.fullname()

class Employee(Person):
    def __init__(self, first_name, last_name, employee_id):
        Person.__init__(self, first_name, last_name)
        self.employee_id = employee_id

def test_employee():
    e = Employee('first', 'last', 1)
    return str(e)

My test of old-style classes was precisely the same, except that I omitted “object” between the parentheses in the class definition of Person.

I used %timeit from within IPython to run the function 100,000 times for each of the two versions (old-style and new-style).  The results surprised me: Old-style classes took 3.09 µs per iteration, while new-style classes took 2.44 µs per iteration, a difference of more than 20 percent!

The bottom line would seem to be that if you’re running large systems in Python and are still using old-style classes, it’s not just worth upgrading to new-style classes for reasons of aesthetics, features, and compatibility.  It’s also going to speed up your code, particularly if you have a large, long-running system that invokes lots of methods.

= and = aren’t equal

When I teach a Ruby or Python class, I always begin by going through the various data types.  My students are typically experienced programmers in Java, C++, or C#, and so it no longer surprises me when I begin to describe numbers, and someone asks, “How many bits is an integer?”

My answer used to be, “Who cares?”  I would then follow this with a demonstration of the fact that in these languages, numbers can be pretty darned big before you have to worry about such things.

But over the last few months, I’ve begun to understand the reason for this question, and others.  Indeed, I have begun to understand one of the reasons why dynamic languages can be so difficult for people to learn after they have worked with a static language.

Let’s take a simple example.  In a typical, C-style statically typed language, you don’t just assign a variable.  You must first declare it with a type.  You can thus say something like this:

int x;
x = 5;

In both Ruby and Python, you can do something similar: 

x = 5    # no type declaration needed

On the face of it, these seem to be doing similar things.  But they aren’t.

In a static language, a variable is an alias to a place in memory.  Thus, when I say “int x”, I’m telling the compiler to set aside an integer-sized piece of memory, and to give it an alias of “x”.  When I say “x = 5”, the compiler will stick the number 5 inside of that int-sized memory location. This is why static languages force you to declare types — so that they can allocate the right amount of space for the data you want to store, and so that they can double-check that the type you’re trying to store won’t overflow that allocated area.

Dynamic languages don’t do this at all.  Whereas assignment in a static language means, “Put the value on the right in the address on the left,” assignment in a dynamic language means, “As of now, the name on the left points to the object on the right.”

In other words, assignment in a dynamic language isn’t really assignment in the traditional sense.  There’s no fixed memory location associated with a variable.  Rather, a variable is just a name in the current scope, pointing to an object.  Given that everything in both Python and Ruby is an object, you never have to worry about assignment not “fitting” into memory.

This is also why you can say “x = 5” and then “x = [1,2,3]” in a dynamic language: Types sit on the data, not on the variable.  As long as a variable is pointing to an object, you’re just fine, because all object pointers are the same size.

The bottom line, then, is that  = in static languages and = in dynamic languages would seem, on the surface, to be doing similar things.  But they’re definitely not.  Once you understand what they are doing — putting data in memory, or telling a name to point to a value — many other mysteries of the language suddenly make more sense.

Ruby and Python and Felix and Oscar

I have been consulting, developing, and offering training classes in both Ruby and Python for a number of years now — more than 15 years in Python, and more than 7 years in Ruby. Inevitably, when someone from one of my courses hears that I use more than one language, they ask me, “So, which one do you prefer?”

One way to address this is to speak like a parent (which I am), and to give the analogy that just like I love all three of my children equally but differently, I love these two languages equally, but differently. But the most recent time I was answering this question, I asked myself, how do you like them differently?  What is appealing about each of these languages?  Why do you enjoy working with (and teaching) both of them?

I began to search for analogies that would describe the relationship between Ruby and Python, and the reason why I enjoy working with them both. I would sometimes extend the children analogy, saying that they’re like siblings.  But beyond the fact that I’m not their parent, I decided that there were enough differences to make the sibling analogy not quite appropriate. Perhaps it would be most appropriate to call them cousins, or even second cousins.

But then I hit upon another analogy, one which might indicate my age and television-watching habits as a child, but which I think is somewhat apt: The Odd Couple.

I remember the Odd Couple as an American sitcom from the 1970s, broadcast in endless reruns on certain stations, in which two divorced men become roommates and friends, despite their with wildly different habits and outlooks on life. (I should note that the Neil Simon play and movie, upon which the TV series was based, is far darker, and really surprised me when I saw it after years of watching the TV show.)

The viewers aren’t ever expected to prefer neatnik, uptight Felix or sloppy, happy-go-lucky Oscar, but rather to appreciate the differences between the two, and to see a bit of themselves in each character.  In some ways — and perhaps more philosophically than was ever intended — the play, movie, and show are there to tell us that there is no one “right” way to approach life, and that each has its advantages and disadvantages.  Balance is the key.

The more I think about it, the more I like this analogy, because it speaks to the differences between the languages, and the reasons why I love to work in each of them. Python, not surprisingly, is Felix: It’s clean, crisp, elegant, and engineered precisely. It’s no surprise that Python has been called “executable psuedo-code,” in that I’ve met a very large number of people (many of whom take my courses) who have been working with Python for months without knowing precisely what they were doing.  

Python is conservative by nature, and that has served the language well for more than two decades.  Indeed, you could argue that the entire 2-to-3 Python upgrade issue, which has been causing ripples of late, is the result of Python betraying this conservative culture, and making a clean break with past versions for the first time in its history. There are parts of Python that drive me crazy, such as len being a builtin function, list.sort not returning a value,  the limits on lambda. the need for both tuples and lists, and the way that super works.  But every language has its issues, and a very large number of them were improved or removed altogether in Python 3.

But other parts of the language are beautiful, such as the way in which operator overloading is done.  Sure, Ruby lets you rewrite + directly, but I think that there’s something about Python’s __add__ which tells newcomers that they should avoid messing with it until they know what they’re doing.  I have also grown to love list comprehensions (as well as dictionary and set comprehensions), even though I readily admit that the syntax is difficult for beginners.   Also, the Python standard library is just a joy to work with; you can really depend on things working pretty well.  And one of the things that people hate at first about Python, namely the required whitespace, is sheer genius in my book.  Decorators are also wonderful; while I don’t use them much, they are an elegant and powerful way to intercept function and class definitions, and do all sorts of wild stuff with them.

Ruby, on the other hand, is Oscar: It’s infinitely flexible, messy, and creative — but it works the way you want it to work.  Ruby inherited many of the characteristics of Perl, which Larry Wall deliberately meant to be close to natural human language.  Sure, it’s a minor miracle that Ruby’s syntax can be described using computers, given its complexity, but that complexity allows me flexibility, creativity, and intellectual excitement that I can’t get elsewhere.  Add blocks to the mixture, and you have a language which gives you raw building blocks that allow you to solve problems quickly, easily, and naturally, with less code than would otherwise be necessary. For example, ActiveRecord might have its problems, but I  generally love its API and the magic that it performs on my behalf.  The way that validations and associations look like declarations (but are actually class methods) is great, making for readable code.

Of course, Ruby has its problems, as well.  The object model is elegant and simple — but nearly impossible for newcomers to the language to grasp.  (I should know, I teach quite a lot of them.)  The fact that everything ends with “end” drives me a bit crazy.  So do the differences between procs, lambdas, and blocks.  And the “stubby lambda” syntax.  But again, every language has its issues and trade-offs, and the ones that Ruby have made are more than reasonable for my work.

Matz has said that Ruby was optimized for programmer happiness, and Avdi Grimm has used the word “joy” to describe programming in Ruby — and I have to agree with both of them. Programming in Python feels like solving a puzzle, but programming in Ruby feels more satisfying; I’m unleashing my creative energies, and using the language to solve problems in the way that I want.  Python is crisp and demanding, and Ruby is messy and fun.  You know, like Felix and Oscar.

Of course, the style of the languages might be very different — but at the end of the day, there’s a lot of overlap between the two.  IPython and Pry, PyPi and RubyGems, dicts and hashes, “def initialize” and “def __init__” — if you know Ruby, then learning Python isn’t very difficult, and vice versa. Both are byte-compiled, interpreted, object-oriented, strongly typed, dynamic languages.  Both have a GIL, which drives people crazy with threading.  Both make reflection and metaprogramming easy and natural. Both encourage modularization of code, with short functions.  Both encourage you to test your code.  Both have active open-source communities.  And both can be used to solve lots of problems, easily and quickly.

Indeed, the languages are similar enough that I’ve often “stolen” ideas, examples, and exercises from my Python classes for my Ruby classes, and vice versa.  And I’ve often thought, when reading the documentation for a method on a built-in Ruby class, that it’s a shame that there’s no Python equivalent… only to discover that there is.

I love Python’s PEP process, which makes it easy for the community to document and discuss changes to the language.  And yet, somehow, Ruby has moved from version 1.9 to 2.0 to 2.1 in the last few years, with great improvement on all fronts, without such a clear-cut process.  I’m not quite sure how Ruby manages to do it, but it does, and rather impressively.

So, which do I prefer?  For Web development, I use Ruby (and Rails or Sinatra).  For small projects and problem solving, and sysadmin types of things, I use Python.  If I had to do large-scale calculations, then NumPy would make Python a no-brainer.  As a first programming language to teach young people, I think that Python is an almost perfect choice.  And for mind-twisting, understand-how-languages-work examples, Ruby beats everyone hands down.

At the end of the day, I’m happy to have a foot in each camp, and to be comfortable with both. Because sometimes you want to be Felix, and sometimes you want to be Oscar, and it’s always nice to have to choose between the two.

Making Python’s __init__ method magical

In some programming languages, the idea of “reflection” is somewhat exotic, and takes a while to learn. In Python (and Ruby, for that matter), the language is both dynamic and open, allowing you to poke around in the internals of just about any object you might like.  Reflection is a natural part of working with these languages, which lends itself to numerous things that you might want to do.

Reflection in Python is easy because everything is an object, and every Python object has attributes, which we can list using “dir”.  For example, I can list the attributes of a string:

>>> s = 'abc'
>>> dir(s)
['__add__', '__class__', '__contains__', '__delattr__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getitem__', '__getnewargs__', '__getslice__', '__gt__', '__hash__', '__init__', '__le__', '__len__', '__lt__', '__mod__', '__mul__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__rmod__', '__rmul__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '_formatter_field_name_split', '_formatter_parser', 'capitalize', 'center', 'count', 'decode', 'encode', 'endswith', 'expandtabs', 'find', 'format', 'index', 'isalnum', 'isalpha', 'isdigit', 'islower', 'isspace', 'istitle', 'isupper', 'join', 'ljust', 'lower', 'lstrip', 'partition', 'replace', 'rfind', 'rindex', 'rjust', 'rpartition', 'rsplit', 'rstrip', 'split', 'splitlines', 'startswith', 'strip', 'swapcase', 'title', 'translate', 'upper', 'zfill']

Since everything is an object, including built-in classes, I can get the attributes of a base type, as well.  For example, I can get the attributes associated with the “str” class:

>>> dir(str)
['__add__', '__class__', '__contains__', '__delattr__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getitem__', '__getnewargs__', '__getslice__', '__gt__', '__hash__', '__init__', '__le__', '__len__', '__lt__', '__mod__', '__mul__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__rmod__', '__rmul__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '_formatter_field_name_split', '_formatter_parser', 'capitalize', 'center', 'count', 'decode', 'encode', 'endswith', 'expandtabs', 'find', 'format', 'index', 'isalnum', 'isalpha', 'isdigit', 'islower', 'isspace', 'istitle', 'isupper', 'join', 'ljust', 'lower', 'lstrip', 'partition', 'replace', 'rfind', 'rindex', 'rjust', 'rpartition', 'rsplit', 'rstrip', 'split', 'splitlines', 'startswith', 'strip', 'swapcase', 'title', 'translate', 'upper', 'zfill']

(If you see a great deal of overlap here between the string instance and the str type, that’s because of the way in which Python handles attribute scoping.  If you cannot find an attribute on an object, then Python looks for the attribute on the object’s class.)

Functions are also objects in Python, which means that we can list their attributes, as well:

def hello(name):
    print "Hello, %s" % name

>>> hello("Dolly")
Hello, Dolly
>>> dir(hello)
['__call__', '__class__', '__closure__', '__code__', '__defaults__', '__delattr__', '__dict__', '__doc__', '__format__', '__get__', '__getattribute__', '__globals__', '__hash__', '__init__', '__module__', '__name__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', 'func_closure', 'func_code', 'func_defaults', 'func_dict', 'func_doc', 'func_globals', 'func_name']

Given an object and the name of one of its attributes, I can retrieve, set, or check for the existence of the attribute value with the builtin “getattr”, “setattr”, and “hasattr” functions.  For example, I can first check to see if an attribute has been defined on an object:

>>> hasattr(hello, 'x')

Then I can set the attribute:

>>> setattr(hello, 'x', 5)
>>> hello.x

Notice that this does indeed mean that I have added a new “x” attribute to my function “hello”. It might sound crazy that Python lets me set an attribute on a function, and even crazier that the attribute can be a random name.  But that’s a fact of life in Python; in almost every case, you can add or change just about any attribute on just about any object. Of course, if you do this to classes that you didn’t create, or that aren’t expecting you to change things, then you can end up causing all sorts of strange behavior.

To retrieve our attribute value, we use the builtin “getattr” function:

>>> getattr(hello, 'x')

Note that there isn’t any difference between saying “hello.x” and invoking getattr.  By the same token, there’s no difference between putting “hello.x” on the left side of an assignment, and using the builtin “setattr” function.

One of the places where we tend to set lots of attributes is in our __init__ methods. While many people think of __init__ as a constructor, that’s not really the case.  Rather, __init__ is invoked on our new object after it has been created; its job is to take the new, blank-slate object and add whatever attributes we’ll want and need during the rest of its life.  It’s thus common for us to do something like this:

class Foo(object):
    def __init__(self, x, y):
        self.x = x
        self.y = y

>>> f = Foo(10, 20)
>>> f.x
>>> f.y

The above is all pretty standard stuff, but let’s still walk through it: We create an instance of Foo.  This means that Python first creates a naked Foo object (using the __new__ method behind the scenes), which is then passes to Foo.__init__, along with the parameter values from our invocation.  Inside of the __init__ method, there are three local variables: self, which points to the new object that we just created, x, and y.  There are actually two x’s and two y’s in this method: The local variables x and y, and the attributes x and y that sit on self.  From Python’s perspective, there is no connection between the local x and self.x.  At the same time, we see an obvious, semantic connection between the two — which is precisely as it should be.

I was teaching a Python class this week, when someone complained to me about the fact that Python code has this repeated boilerplate code in each object’s __init__ method.  Why, he asked me, does Python not have a way to automatically take all of the local variables, and pass them directly as attributes on the instance?  That is, if I pass parameters ‘x’ and ‘y’ to my object instantiation, then the method shouldn’t have to be told to run self.x = x, or self.y = y.  Rather, Python could just do that automatically

This is almost certainly not something that we want to do in Python.  The language (and its community) loves to make things explicit rather than implicit (as stated in the Zen of Python), and this sort of black magic seems more appropriate for Ruby than Python.  But it did seem like an interesting challenge to find out how easily I could implement such a thing, and I gladly took the challenge.

In order to solve this problem, I decided to work backwards: My ultimate goal would be to set attributes on self. That is, I would want to invoke setattr on self, passing it the name of the attribute I want to set, plus the associated value.  Thus, if the function sees that it has parameters self, x, and y, it should invoke setattr(self, ‘x’, VALUE) and setattr(self, ‘y’, VALUE).  I say VALUE here, because we still haven’t figured out where we’ll get the attribute names from, let alone their values.

It’s actually not that hard, as a general rule, to get the attribute names from a function.  I can just go to any function and ask for the __code__ attribute. Underneath that, among other things, is a co_varnames attribute, containing the names of local variables defined in the function.  So if I can just get __code__.co_varnames from inside of __init__ when it is invoked, we’ll be in great shape. (Note that in Python 3, the names of these attributes have changed slightly.)

Well, sort of: It’s nice that __code__ is available as an attribute on the function, but those attributes are only available via the function’s name.  How can I refer to the function from within the function itself?  Is there a pointer to “the current function”?

Not directly, no.  But the “inspect” module, which comes with Python, is perfect for such cases.  Normally, we think of inspect as allowing us to look at Python objects from the outside.  But inspect also allows us to look at the current stack frame, which includes the function that is currently executing.  For example:

>>>frame = inspect.currentframe()
>>> frame.f_code
<code object <module> at 0x106a24830, file "<stdin>", line 1>

The above was executed outside of a function, which means that the function-related information is missing.  Things get much more interesting when we’re inside of a function, however: f_code returns the function object on which we’re working (i.e., the stuff that is usually under the function’s __code__ attribute):

def foo():

>>> foo()

We can also get other function attributes, such as the names of local variables:

def foo(x):

>>> foo(5)

As you can see, we can get the names of local variables — including parameter names — from the co_varnames attribute. A very simple version of our “autoinit” function could thus take an object and one or more parameters, the names of which would be used to set attributes on that object. For example:

def autoinit(obj, x, y):
    for varname in inspect.currentframe().f_code.co_varnames:
        if varname == 'obj':
            setattr(obj, varname, 'hello')

>>> import os
>>> autoinit(os, 5, 10)
>>> os.x
>>> os.y

In the above example, we define our function such that it’ll take the names of all local variables (except for “obj”) and assign new attributes to that object.   However, the value is always ‘hello’.  How can we assign the actual values that are being passed to the parameters?

The easiest solution, I think, is to use the locals() function, which returns a dictionary of the currently defined local variables.  The keys of this dictionary are strings, which means that we can pretty trivially use the variable names to grab the value — and then make the assignment:

def autoinit(obj, x, y):
    for varname in inspect.currentframe().f_code.co_varnames:
        if varname == 'obj':
            setattr(obj, varname, locals()[varname])

>>> autoinit(os, 5, 10)
>>> os.x
>>> os.y

So far, so good!  We now have a function that can use the parameter names in setting attributes.  However, if this function is going to work, then it’s not going to exist on its own. Rather, we want to invoke “autoinit” from within our __init__ method.  This means that we need autoinit to set attributes not based on its own parameters, but rather based on its caller’s parameters.  The inspect.currentframe method returns the current stack frame, but we want the caller’s stack frame.

Fortunately, the implementers of inspect.currentframe thought of this, and provide an easy and elegant solution: If we invoke inspect.currentframe with a numeric parameter, Python will jump back that number of stack frames.  Thus, inspect.currentframe() returns the current stack frame, inspect.currentframe(1) returns the caller’s stack frame, and inspect.currentframe(2) inspect the caller’s caller’s stack frame.

By invoking inspect.currentframe(1) from within __init__, we can get the instance onto which we want to add the attributes, as well as the attribute names and values themselves. For example:

import inspect
def autoinit():
    frame = inspect.currentframe(1)
    params = frame.f_locals
    self = params['self']
    paramnames = frame.f_code.co_varnames[1:] # ignore self
    for name in paramnames:
        setattr(self, name, params[name])

class Foo(object):
    def __init__(self, x, y):

>>> f = Foo(100, 'abc')
>>> f.x
>>> f.y
>>> g = Foo(200, 'ghi')
>>> g.x
>>> g.y

Hey, that’s looking pretty good!  However, there is still one problem: Python doesn’t see a difference between parameters and local variables.  This means that if we create a local variable within __init__, autoinit will get confused:

class Bar(object):
    def __init__(self, x, y):
        z = 999
>>> b = Bar(100, 'xyz')
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
 File "<stdin>", line 3, in __init__
 File "<stdin>", line 7, in autoinit
KeyError: 'z'

As you can see, autoinit tries to find the value of our ‘z’ local variable — but because the variable has not been set, it isn’t in the locals() dictionary.  But the bigger problem is that autoinit is trying to look for z at all — as a local variable, rather than a parameter, we don’t want it to be set as an attribute on self!

The solution is to use the co_argcount attribute to our code object, which says how many arguments our function takes. For example:

def foo(x):
    y = 100
    return inspect.currentframe()

>>> s = foo(5)
>>> print(s.f_code.co_argcount)
>>> print(s.f_code.co_varnames)
('x', 'y')

We can improve our implementation of autoinit by only taking the first co_argcount elements of co_varnames. So far as I can tell (and I don’t know if this is official, or just a convenient accident), the arguments always come first in co_varnames.  Our final version of autoinit thus looks like this:

def autoinit():
    frame = inspect.currentframe(1)
    params = frame.f_locals
    nparams = frame.f_code.co_argcount
    self = params['self']
    paramnames = frame.f_code.co_varnames[1:nparams]
    for name in paramnames:
        setattr(self, name, params[name]) 

Sure enough, if we try it:

class Foo(object):
    def __init__(self, x, y):
        z = 100
>>> f = Foo(10, 20)
>>> print f.x
>>> print f.y

Success!  Of course, this implementation still doesn’t handle *args or **kwargs.  And as I wrote above, it’s very much not in the spirit of Python to have such magic happening behind the scenes.  Yet, I’ve found this to be an interesting way to discover and play with functions, the “inspect” module, and how arguments are handled.

If you liked this explanation, then you’ll likely also enjoy my ebook, “Practice Makes Python,” with 50 exercises meant to improve your Python fluency.

Python scoping e-mail course

I had so much fun writing the previous blog post about Python scoping that I decided to expand it into a free e-mail course.  Each day (for five days), you’ll receive another lesson about how scopes work in Python, and why this is important for you to know as a Python developer.

So if you’ve ever been unclear on the “LEGB rule,” wanted to know when and how to use the “global keyword,” or even how you can nearly break your Python implementation through scope abuse, this e-mail course should help.

Please e-mail me if you have questions or comments about this e-mail course!  I’ve had so much fun putting this one together that I’m very likely to create additional ones, so your suggestions for future topics are extremely welcome.