rvm do

I’ve been using rvm for many years, and love it.  Yes, I know that it rewrites simple commands, such as “ruby” and “gem”, so that I can use lots of different Ruby versions.  Yes, I know that it can be overkill for certain situations.  And yes, I know that rbenv is preferred by many.

But I’ve been using rvm for a long time, and I find it works very well for my needs.  I can (and do) have many different versions of Ruby running on my computer, and having access to all of them at once is terrific.

I tend to be obsessive about updating Ruby gems on my systems, and I’m sure I’m not the only Rubyist who runs “gem update -V” (and yes, I love the “verbose” option) at least once per day.  Updating gems never removes the old versions, and if you’re using Bundler in your Rails or Sinatra application, then it doesn’t really matter how many versions you have on your system.  (And yes, I know that willy-nilly updating all gems on my system is probably not wise.  If only that were the most foolish thing I do…)

The thing is, when I update gems, I do so in a particular version of Ruby.  So even though I’m always running “gem update -V”, I never quite remember which versions of Ruby have the latest gems, and which haven’t been updated in a while.  There is, of course, a clear correlation between the frequency with which I use a Ruby version and the freshness of the gems for that version on my system.  But I sometimes find myself having to update gems in a version of Ruby that I haven’t used in a while.

So you can imagine my delight when I discovered “rvm do”.  This is an rvm command that lets you execute a command in any or all of the Ruby versions installed on your system.  It basically switches to the Ruby version and then executes the requested shell command — so you’re not executing a Ruby program in each separate version, but rather you’re executing a command-line program once for each version of Ruby installed.  You can think of it as executing the same shell command once for each installed version, prefaced by “rvm VERSION_NUMBER”.

So, how can I ensure that all of the gems, for all versions of Ruby, are up to date?  Very simply, I write:

rvm all do gem update -V

And if I want to check out some Ruby code, and see how it runs in all of the versions on my system, I can say

rvm all do ruby test.rb

If I just want to see the difference between doing something in 1.8, 1.9, 2.0, and 2.1 (without all of the patchlevels for 1.9.3), then I can just say:

rvm 1.8.7,1.9.3,2.0,2.1  do ruby test.rb

I’m already loving this feature, and can easily imagine cases — such as when teaching Ruby programming, and trying to show them the differences between versions — when this will be quite handy.

Announcing: Teaching to Code, a community of programming instructors

In the wake of my last blog post, I’ve been thinking a great deal about the practice of teaching, and specifically the practice of teaching programming.  I’ve realized that while instruction in programming is increasingly popular and important, the people engaged in such instruction aren’t comparing notes, learning from one another, or generally working to improve the trade.

I’ve decided to try to change that.  I’ve created a new site, Teaching to Code, a discussion forum aimed at anyone who teaches programming to others.  Whether you teach in person, produce screencasts, or lecture at the university level, I’m sure that there are techniques, ideas, and suggestions that you can share with other people, and which can help to improve the craft of teaching programming.

It’s true that many of us in this community are commercial instructors.  As a result, there will undoubtedly be some overlap and competition among the people who participate.  I’m optimistic that we can balance these competitive instincts and realities with the goal that we all (presumably) have, namely to improve our students’ knowledge and understanding of programming in general, and of the technologies we teach in particular.

In addition to general discussion on a variety of topics, I’m also aiming to have a monthly book/journal club.  Each month, we’ll discuss a book, journal article, or blog post (or even a video, I guess) that can inform and improve our teaching.  Some of the initial suggestions will come from readings I’ve had in graduate school; there were a number of papers that have really influenced my thinking, and that I believe will be interesting and useful for others, too.  But I know that I’ve only read a minority of things written on this subject, and would be delighted to read and then discuss these items, as well.

If you’re a programming instructor of any sort, please join us!  Contribute to the fledgling discussion, and suggest how we can make it better.  If there is something that you feel could help you, or improve your teaching, then you can either ask on the forum or e-mail me at reuven@lerner.co.il.  Either way, I hope that Teaching to Code will become a community of practice for programming instructors worldwide, helping teachers and students alike.

Teaching and acting (or, why I don’t plan to sell recorded classes in the near future)

Several weeks ago, my wife and I saw a wonderful play at our local theater in Modi’in  (“Mother Courage and Her Children“).  At the end, the actors came out to receive their richly deserved applause.  Three times, the actors came out, took their bows, and were warmly applauded by the audience.  We loved their performance — but just as importantly, they loved performing, and they loved to see and hear the reactions from the audience, both during and after the play.

I’m sure that some or all of these actors have worked in television and the movies; Israel is a small country, and it’s hard for me to believe that actors can decide only to work in a single medium.  But I’ve often heard that actors prefer to work on stage, because they can have a connection with the audience.  When they say something funny, sad, or upsetting, they can feel (and even hear) the audience’s reaction.

But while we often hear about TV and movie stars making many millions of dollars off of their work, it’s less common for stage actors to make that kind of money.  That’s because when you act on stage, you’re by definition limiting your audience to the number of people who can fit in a theater.  Even the largest theaters aren’t going to hold more than a few hundred seats; by contrast, even a semi-successful TV show or movie will get tens or hundreds of thousands of viewers on a given night.  (And yes, TV and film have many more expenses than plays do — but the fact remains that you can scale up the number of TV and film viewers much more easily than you can a play.  Plus, movies and TV can both be shown in reruns.)

Another difference is the effort that you need to put into a stage production, as opposed to a TV program or a movie: In the former case, you need to perform each and every night.  In the latter, you record your performance once — and yes, it’ll probably require multiple takes — and then it can be shown any number of times in the future.  You can even be acting on stage while your TV show is broadcast.  Or more than one of your movies can be shown simultaneously, in thousands of cities around the world.

What does this have to do with me?  And why have I been thinking about this so much over the last few weeks, since seeing that play?

While I’m a software developer and consultant, I also spend a not-insignificant time teaching people: In any given week, I will give 2-4 full days of classes in Python, Ruby, Ruby on Rails, PostgreSQL, and Git, with other classes likely to come in the next few months.

I’m starting to dip my toes into the waters of teaching online, and hope to do it increasingly frequently over the coming months and years.  But unlike most online programming courses currently being offered, I intend to make most or all of my courses real-time, live, and in person.

This has some obvious disadvantages: It means that people will need to be available during the precise hours that I’m teaching. It means that the course will have to be higher in price than a pre-recorded video course, because I cannot amortize my time investment over many different purchases and viewings.  And it means that the course is limited in size; I cannot imagine teaching more than 10 people online, just as I won’t teach an in-person class with more than 20 people.

Given all of these disadvantages, why would I prefer to do things this way, live and in person?

The answer, in a word, is: Interactions.

I’m finishing my PhD in Learning Sciences, and if there’s anything that I have gained from my studies and research, it’s that personal interactions are the key to deep learning. That’s why my research is all about online collaboration; I deeply believe that it’s easiest and best to learn when you speak with, ask questions of, challenge, and collaborate with others, ideally when you’re trying to solve a problem.

I’m not saying that it’s impossible to learn on your own; I certainly spend enough hours each week watching screencasts and lectures, and reading blog posts, to demonstrate that it’s possible, pleasurable, and beneficial to learn in these ways. But if you want to understand a subject deeply, then you should communicate somehow with other people.

That’s one of the reasons why pair programming is so helpful, improving both the resulting software and the programmers who engage in the pairing. That’s why open source is so successful — because in a high-quality open-source project, you’ll have people constantly interacting, discussing, arguing, and finally agreeing on the best way to do things. And that’s why I constantly encourage participants in my classes to work together when they’re working on the exercises that I ask them to solve: Talking to someone else will help you to learn better, more quickly, and more deeply.

I thus believe that attending an in-person class offers many advantages over seeing a recorded screencast or lecture, not because the content is necessarily better, but because you have the opportunity to ask questions, to interact with the teacher, to clarify points that weren’t obvious the first time around, and to ask how you might be able to integrate the lectures into your existing work environment.

So for the students, an in-person class is a huge win.  What do I get out of it?  Why do I prefer to teach in person?

To answer that, I return to the topic with which I started this post, namely actors who prefer to work on stage, rather than on TV and in movies. When I give a course, it’s almost like I’m putting on a one-man show. Just as actors can give the same performance night after night without getting bored, I can give the same “introduction to Python” course dozens of times a year without tiring of it.  (And yes, I do constantly update my course materials — but even so, the class has stayed largely the same for some time.)  I’m putting on a show, albeit an interactive and educational one, and while I put on the same show time after time, I don’t get tired of it.

And the reason that I don’t get tired of it? Those same interactions, which are so beneficial to the students’ learning and progress, are good for me, as the instructor.  They keep me on my toes, allow me to know what is working (and what isn’t), provide me with an opportunity to dive more deeply into a subject that is of particular interest to the participants, and assure me that the topics I’m covering are useful and important for the people taking my class.

I live and work in Israel, and one of the things that I love about teaching Israelis is that I’m almost guaranteed to be challenged and questioned at nearly ever turn. Israelis are, by nature, antagonistic toward authority.  As a result, my lectures are constantly interrupted by questions, challenges, and requests for proof.

I have grown so accustomed to this way of things, that it once backfired on me: Years ago, I gave a one-day course in the US that ended at lunchtime — it turns out that the Americans were very polite and quiet, and didn’t ask any questions, allowing me to get through an entire day’s worth of material in just half of the time.  I have since learned to make cultural adjustments to the number of slides I prepare for a given day, depending on where I will be teaching!

When I look at stage actors, and see them giving the same performance that they have given an untold number of times in the past, I now understand where they’re coming from. For them, each night gives them a chance to expose a new audience to the ideas that they’re trying to get across through their characters and dialogue.  And yes, they could do that in a movie — but then they would be missing the interactions that they have with the audience, which provide a sense of excitement that’s hard to match.

Does this mean that I won’t ever record screencasts or lectures?  No, I’m sure that I will do that at some point, and I already have some ideas for doing so. But they’ll be fundamentally different from the courses that I teach, complementing the full-length courses, rather than replacing them. At the end of the day, I get a great deal of satisfaction from lecturing and teaching, both because I see that people are learning (and thus gaining a useful skill), and because my interactions with them are so precious to me, as an instructor.

Benchmarking old-style and new-style Python classes

It has been many years since Python developers were really supposed to worry about new-style vs. old-style classes.  There is only one style (new) in Python 3.x, and even in Python 2.x, old-style classes have not been recommended for many years.  Nevertheless, I mention old-style classes in my Python courses, mostly so that participants will understand the potentially serious implications of creating classes without inheriting from object.  For example:

>>> class Foo(object): pass
>>> type(Foo)
type

>>> f = Foo()
>>> type(f)
__main__.Foo

The above is the way that modern Python programmers define classes.  This is the preferred way, for sure; if you’re writing old-style classes, then you’re almost certainly doing something wrong.  But it’s so easy to create an old-style class in Python — all you have to do is forget to inherit from “object”:

>>> class Foo(): pass
>>> type(Foo)
classobj

>>> f = Foo()
>>> type(f)
instance

As you can see, the fact that I created an old-style class directly affects the types of objects that I have created, and thus their capabilities.  For many years, it has been seen as a mistake to create old-style classes; not only are you missing out on new functionality, but you are creating objects that behave differently from the rest of objects in Python.

I was just teaching a Python class at a company that has a fair amount of legacy Python code.  It turns out that this legacy code includes a large number of old-style classes. The company asked me whether it was worth upgrading all of their old-style classes to use new-style classes; my answer was that (1) if it ain’t broke, don’t fix it, (2) it’s hard to know whether the upgrade would be trivially easy or impossibly hard, and (3) you’ll likely want to upgrade these classes over time, doing so incrementally.

Someone then asked me whether there is a performance difference between old-style and new-style classes, in order to evaluate the importance of doing such an upgrade project.  I had to admit that I wasn’t sure, and couldn’t find anything online (after doing a quick search) on the subject.  I thus decided to do a small benchmark to see what might be faster (or slower).  I’m not an expert in benchmarking, but I did want to check the basic speed of (1) object creation, (2) inheritance, and (3) implementation of __repr__.

The results surprised me: New-style classes are substantially faster.  Here is the benchmark that I ran on the new-style class:

class Person(object):
    def __init__(self, first_name, last_name):
        self.first_name = first_name
        self.last_name = last_name
    def fullname(self):
        return self.first_name + " " + self.last_name
    def __repr__(self):
        return self.fullname()

class Employee(Person):
    def __init__(self, first_name, last_name, employee_id):
        Person.__init__(self, first_name, last_name)
        self.employee_id = employee_id

def test_employee():
    e = Employee('first', 'last', 1)
    return str(e)

My test of old-style classes was precisely the same, except that I omitted “object” between the parentheses in the class definition of Person.

I used %timeit from within IPython to run the function 100,000 times for each of the two versions (old-style and new-style).  The results surprised me: Old-style classes took 3.09 µs per iteration, while new-style classes took 2.44 µs per iteration, a difference of more than 20 percent!

The bottom line would seem to be that if you’re running large systems in Python and are still using old-style classes, it’s not just worth upgrading to new-style classes for reasons of aesthetics, features, and compatibility.  It’s also going to speed up your code, particularly if you have a large, long-running system that invokes lots of methods.

= and = aren’t equal

When I teach a Ruby or Python class, I always begin by going through the various data types.  My students are typically experienced programmers in Java, C++, or C#, and so it no longer surprises me when I begin to describe numbers, and someone asks, “How many bits is an integer?”

My answer used to be, “Who cares?”  I would then follow this with a demonstration of the fact that in these languages, numbers can be pretty darned big before you have to worry about such things.

But over the last few months, I’ve begun to understand the reason for this question, and others.  Indeed, I have begun to understand one of the reasons why dynamic languages can be so difficult for people to learn after they have worked with a static language.

Let’s take a simple example.  In a typical, C-style statically typed language, you don’t just assign a variable.  You must first declare it with a type.  You can thus say something like this:

int x;
x = 5;

In both Ruby and Python, you can do something similar: 

x = 5    # no type declaration needed

On the face of it, these seem to be doing similar things.  But they aren’t.

In a static language, a variable is an alias to a place in memory.  Thus, when I say “int x”, I’m telling the compiler to set aside an integer-sized piece of memory, and to give it an alias of “x”.  When I say “x = 5″, the compiler will stick the number 5 inside of that int-sized memory location. This is why static languages force you to declare types — so that they can allocate the right amount of space for the data you want to store, and so that they can double-check that the type you’re trying to store won’t overflow that allocated area.

Dynamic languages don’t do this at all.  Whereas assignment in a static language means, “Put the value on the right in the address on the left,” assignment in a dynamic language means, “As of now, the name on the left points to the object on the right.”

In other words, assignment in a dynamic language isn’t really assignment in the traditional sense.  There’s no fixed memory location associated with a variable.  Rather, a variable is just a name in the current scope, pointing to an object.  Given that everything in both Python and Ruby is an object, you never have to worry about assignment not “fitting” into memory.

This is also why you can say “x = 5″ and then “x = [1,2,3]” in a dynamic language: Types sit on the data, not on the variable.  As long as a variable is pointing to an object, you’re just fine, because all object pointers are the same size.

The bottom line, then, is that  = in static languages and = in dynamic languages would seem, on the surface, to be doing similar things.  But they’re definitely not.  Once you understand what they are doing — putting data in memory, or telling a name to point to a value — many other mysteries of the language suddenly make more sense.

Ruby and Python and Felix and Oscar

I have been consulting, developing, and offering training classes in both Ruby and Python for a number of years now — more than 15 years in Python, and more than 7 years in Ruby. Inevitably, when someone from one of my courses hears that I use more than one language, they ask me, “So, which one do you prefer?”

One way to address this is to speak like a parent (which I am), and to give the analogy that just like I love all three of my children equally but differently, I love these two languages equally, but differently. But the most recent time I was answering this question, I asked myself, how do you like them differently?  What is appealing about each of these languages?  Why do you enjoy working with (and teaching) both of them?

I began to search for analogies that would describe the relationship between Ruby and Python, and the reason why I enjoy working with them both. I would sometimes extend the children analogy, saying that they’re like siblings.  But beyond the fact that I’m not their parent, I decided that there were enough differences to make the sibling analogy not quite appropriate. Perhaps it would be most appropriate to call them cousins, or even second cousins.

But then I hit upon another analogy, one which might indicate my age and television-watching habits as a child, but which I think is somewhat apt: The Odd Couple.

I remember the Odd Couple as an American sitcom from the 1970s, broadcast in endless reruns on certain stations, in which two divorced men become roommates and friends, despite their with wildly different habits and outlooks on life. (I should note that the Neil Simon play and movie, upon which the TV series was based, is far darker, and really surprised me when I saw it after years of watching the TV show.)

The viewers aren’t ever expected to prefer neatnik, uptight Felix or sloppy, happy-go-lucky Oscar, but rather to appreciate the differences between the two, and to see a bit of themselves in each character.  In some ways — and perhaps more philosophically than was ever intended — the play, movie, and show are there to tell us that there is no one “right” way to approach life, and that each has its advantages and disadvantages.  Balance is the key.

The more I think about it, the more I like this analogy, because it speaks to the differences between the languages, and the reasons why I love to work in each of them. Python, not surprisingly, is Felix: It’s clean, crisp, elegant, and engineered precisely. It’s no surprise that Python has been called “executable psuedo-code,” in that I’ve met a very large number of people (many of whom take my courses) who have been working with Python for months without knowing precisely what they were doing.  

Python is conservative by nature, and that has served the language well for more than two decades.  Indeed, you could argue that the entire 2-to-3 Python upgrade issue, which has been causing ripples of late, is the result of Python betraying this conservative culture, and making a clean break with past versions for the first time in its history. There are parts of Python that drive me crazy, such as len being a builtin function, list.sort not returning a value,  the limits on lambda. the need for both tuples and lists, and the way that super works.  But every language has its issues, and a very large number of them were improved or removed altogether in Python 3.

But other parts of the language are beautiful, such as the way in which operator overloading is done.  Sure, Ruby lets you rewrite + directly, but I think that there’s something about Python’s __add__ which tells newcomers that they should avoid messing with it until they know what they’re doing.  I have also grown to love list comprehensions (as well as dictionary and set comprehensions), even though I readily admit that the syntax is difficult for beginners.   Also, the Python standard library is just a joy to work with; you can really depend on things working pretty well.  And one of the things that people hate at first about Python, namely the required whitespace, is sheer genius in my book.  Decorators are also wonderful; while I don’t use them much, they are an elegant and powerful way to intercept function and class definitions, and do all sorts of wild stuff with them.

Ruby, on the other hand, is Oscar: It’s infinitely flexible, messy, and creative — but it works the way you want it to work.  Ruby inherited many of the characteristics of Perl, which Larry Wall deliberately meant to be close to natural human language.  Sure, it’s a minor miracle that Ruby’s syntax can be described using computers, given its complexity, but that complexity allows me flexibility, creativity, and intellectual excitement that I can’t get elsewhere.  Add blocks to the mixture, and you have a language which gives you raw building blocks that allow you to solve problems quickly, easily, and naturally, with less code than would otherwise be necessary. For example, ActiveRecord might have its problems, but I  generally love its API and the magic that it performs on my behalf.  The way that validations and associations look like declarations (but are actually class methods) is great, making for readable code.

Of course, Ruby has its problems, as well.  The object model is elegant and simple — but nearly impossible for newcomers to the language to grasp.  (I should know, I teach quite a lot of them.)  The fact that everything ends with “end” drives me a bit crazy.  So do the differences between procs, lambdas, and blocks.  And the “stubby lambda” syntax.  But again, every language has its issues and trade-offs, and the ones that Ruby have made are more than reasonable for my work.

Matz has said that Ruby was optimized for programmer happiness, and Avdi Grimm has used the word “joy” to describe programming in Ruby — and I have to agree with both of them. Programming in Python feels like solving a puzzle, but programming in Ruby feels more satisfying; I’m unleashing my creative energies, and using the language to solve problems in the way that I want.  Python is crisp and demanding, and Ruby is messy and fun.  You know, like Felix and Oscar.

Of course, the style of the languages might be very different — but at the end of the day, there’s a lot of overlap between the two.  IPython and Pry, PyPi and RubyGems, dicts and hashes, “def initialize” and “def __init__” — if you know Ruby, then learning Python isn’t very difficult, and vice versa. Both are byte-compiled, interpreted, object-oriented, strongly typed, dynamic languages.  Both have a GIL, which drives people crazy with threading.  Both make reflection and metaprogramming easy and natural. Both encourage modularization of code, with short functions.  Both encourage you to test your code.  Both have active open-source communities.  And both can be used to solve lots of problems, easily and quickly.

Indeed, the languages are similar enough that I’ve often “stolen” ideas, examples, and exercises from my Python classes for my Ruby classes, and vice versa.  And I’ve often thought, when reading the documentation for a method on a built-in Ruby class, that it’s a shame that there’s no Python equivalent… only to discover that there is.

I love Python’s PEP process, which makes it easy for the community to document and discuss changes to the language.  And yet, somehow, Ruby has moved from version 1.9 to 2.0 to 2.1 in the last few years, with great improvement on all fronts, without such a clear-cut process.  I’m not quite sure how Ruby manages to do it, but it does, and rather impressively.

So, which do I prefer?  For Web development, I use Ruby (and Rails or Sinatra).  For small projects and problem solving, and sysadmin types of things, I use Python.  If I had to do large-scale calculations, then NumPy would make Python a no-brainer.  As a first programming language to teach young people, I think that Python is an almost perfect choice.  And for mind-twisting, understand-how-languages-work examples, Ruby beats everyone hands down.

At the end of the day, I’m happy to have a foot in each camp, and to be comfortable with both. Because sometimes you want to be Felix, and sometimes you want to be Oscar, and it’s always nice to have to choose between the two.

Making Python’s __init__ method magical

In some programming languages, the idea of “reflection” is somewhat exotic, and takes a while to learn. In Python (and Ruby, for that matter), the language is both dynamic and open, allowing you to poke around in the internals of just about any object you might like.  Reflection is a natural part of working with these languages, which lends itself to numerous things that you might want to do.

Reflection in Python is easy because everything is an object, and every Python object has attributes, which we can list using “dir”.  For example, I can list the attributes of a string:

>>> s = 'abc'
>>> dir(s)
['__add__', '__class__', '__contains__', '__delattr__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getitem__', '__getnewargs__', '__getslice__', '__gt__', '__hash__', '__init__', '__le__', '__len__', '__lt__', '__mod__', '__mul__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__rmod__', '__rmul__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '_formatter_field_name_split', '_formatter_parser', 'capitalize', 'center', 'count', 'decode', 'encode', 'endswith', 'expandtabs', 'find', 'format', 'index', 'isalnum', 'isalpha', 'isdigit', 'islower', 'isspace', 'istitle', 'isupper', 'join', 'ljust', 'lower', 'lstrip', 'partition', 'replace', 'rfind', 'rindex', 'rjust', 'rpartition', 'rsplit', 'rstrip', 'split', 'splitlines', 'startswith', 'strip', 'swapcase', 'title', 'translate', 'upper', 'zfill']

Since everything is an object, including built-in classes, I can get the attributes of a base type, as well.  For example, I can get the attributes associated with the “str” class:

>>> dir(str)
['__add__', '__class__', '__contains__', '__delattr__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getitem__', '__getnewargs__', '__getslice__', '__gt__', '__hash__', '__init__', '__le__', '__len__', '__lt__', '__mod__', '__mul__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__rmod__', '__rmul__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '_formatter_field_name_split', '_formatter_parser', 'capitalize', 'center', 'count', 'decode', 'encode', 'endswith', 'expandtabs', 'find', 'format', 'index', 'isalnum', 'isalpha', 'isdigit', 'islower', 'isspace', 'istitle', 'isupper', 'join', 'ljust', 'lower', 'lstrip', 'partition', 'replace', 'rfind', 'rindex', 'rjust', 'rpartition', 'rsplit', 'rstrip', 'split', 'splitlines', 'startswith', 'strip', 'swapcase', 'title', 'translate', 'upper', 'zfill']

(If you see a great deal of overlap here between the string instance and the str type, that’s because of the way in which Python handles attribute scoping.  If you cannot find an attribute on an object, then Python looks for the attribute on the object’s class.)

Functions are also objects in Python, which means that we can list their attributes, as well:

def hello(name):
    print "Hello, %s" % name

>>> hello("Dolly")
Hello, Dolly
>>> dir(hello)
['__call__', '__class__', '__closure__', '__code__', '__defaults__', '__delattr__', '__dict__', '__doc__', '__format__', '__get__', '__getattribute__', '__globals__', '__hash__', '__init__', '__module__', '__name__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', 'func_closure', 'func_code', 'func_defaults', 'func_dict', 'func_doc', 'func_globals', 'func_name']

Given an object and the name of one of its attributes, I can retrieve, set, or check for the existence of the attribute value with the builtin “getattr”, “setattr”, and “hasattr” functions.  For example, I can first check to see if an attribute has been defined on an object:

>>> hasattr(hello, 'x')
False

Then I can set the attribute:

>>> setattr(hello, 'x', 5)
>>> hello.x
5

Notice that this does indeed mean that I have added a new “x” attribute to my function “hello”. It might sound crazy that Python lets me set an attribute on a function, and even crazier that the attribute can be a random name.  But that’s a fact of life in Python; in almost every case, you can add or change just about any attribute on just about any object. Of course, if you do this to classes that you didn’t create, or that aren’t expecting you to change things, then you can end up causing all sorts of strange behavior.

To retrieve our attribute value, we use the builtin “getattr” function:

>>> getattr(hello, 'x')
5

Note that there isn’t any difference between saying “hello.x” and invoking getattr.  By the same token, there’s no difference between putting “hello.x” on the left side of an assignment, and using the builtin “setattr” function.

One of the places where we tend to set lots of attributes is in our __init__ methods. While many people think of __init__ as a constructor, that’s not really the case.  Rather, __init__ is invoked on our new object after it has been created; its job is to take the new, blank-slate object and add whatever attributes we’ll want and need during the rest of its life.  It’s thus common for us to do something like this:

class Foo(object):
    def __init__(self, x, y):
        self.x = x
        self.y = y

>>> f = Foo(10, 20)
>>> f.x
10
>>> f.y
20

The above is all pretty standard stuff, but let’s still walk through it: We create an instance of Foo.  This means that Python first creates a naked Foo object (using the __new__ method behind the scenes), which is then passes to Foo.__init__, along with the parameter values from our invocation.  Inside of the __init__ method, there are three local variables: self, which points to the new object that we just created, x, and y.  There are actually two x’s and two y’s in this method: The local variables x and y, and the attributes x and y that sit on self.  From Python’s perspective, there is no connection between the local x and self.x.  At the same time, we see an obvious, semantic connection between the two — which is precisely as it should be.

I was teaching a Python class this week, when someone complained to me about the fact that Python code has this repeated boilerplate code in each object’s __init__ method.  Why, he asked me, does Python not have a way to automatically take all of the local variables, and pass them directly as attributes on the instance?  That is, if I pass parameters ‘x’ and ‘y’ to my object instantiation, then the method shouldn’t have to be told to run self.x = x, or self.y = y.  Rather, Python could just do that automatically

This is almost certainly not something that we want to do in Python.  The language (and its community) loves to make things explicit rather than implicit (as stated in the Zen of Python), and this sort of black magic seems more appropriate for Ruby than Python.  But it did seem like an interesting challenge to find out how easily I could implement such a thing, and I gladly took the challenge.

In order to solve this problem, I decided to work backwards: My ultimate goal would be to set attributes on self. That is, I would want to invoke setattr on self, passing it the name of the attribute I want to set, plus the associated value.  Thus, if the function sees that it has parameters self, x, and y, it should invoke setattr(self, ‘x’, VALUE) and setattr(self, ‘y’, VALUE).  I say VALUE here, because we still haven’t figured out where we’ll get the attribute names from, let alone their values.

It’s actually not that hard, as a general rule, to get the attribute names from a function.  I can just go to any function and ask for the __code__ attribute. Underneath that, among other things, is a co_varnames attribute, containing the names of local variables defined in the function.  So if I can just get __code__.co_varnames from inside of __init__ when it is invoked, we’ll be in great shape. (Note that in Python 3, the names of these attributes have changed slightly.)

Well, sort of: It’s nice that __code__ is available as an attribute on the function, but those attributes are only available via the function’s name.  How can I refer to the function from within the function itself?  Is there a pointer to “the current function”?

Not directly, no.  But the “inspect” module, which comes with Python, is perfect for such cases.  Normally, we think of inspect as allowing us to look at Python objects from the outside.  But inspect also allows us to look at the current stack frame, which includes the function that is currently executing.  For example:

>>>frame = inspect.currentframe()
>>> frame.f_code
<code object <module> at 0x106a24830, file "<stdin>", line 1>

The above was executed outside of a function, which means that the function-related information is missing.  Things get much more interesting when we’re inside of a function, however: f_code returns the function object on which we’re working (i.e., the stuff that is usually under the function’s __code__ attribute):

def foo():
    print(inspect.currentframe().f_code.co_name)

>>> foo()
foo

We can also get other function attributes, such as the names of local variables:

def foo(x):
    print(inspect.currentframe().f_code.co_varnames)

>>> foo(5)
('x',)

As you can see, we can get the names of local variables — including parameter names — from the co_varnames attribute. A very simple version of our “autoinit” function could thus take an object and one or more parameters, the names of which would be used to set attributes on that object. For example:

def autoinit(obj, x, y):
    for varname in inspect.currentframe().f_code.co_varnames:
        if varname == 'obj':
            continue
        else:
            setattr(obj, varname, 'hello')

>>> import os
>>> autoinit(os, 5, 10)
>>> os.x
'hello'
>>> os.y
'hello'

In the above example, we define our function such that it’ll take the names of all local variables (except for “obj”) and assign new attributes to that object.   However, the value is always ‘hello’.  How can we assign the actual values that are being passed to the parameters?

The easiest solution, I think, is to use the locals() function, which returns a dictionary of the currently defined local variables.  The keys of this dictionary are strings, which means that we can pretty trivially use the variable names to grab the value — and then make the assignment:

def autoinit(obj, x, y):
    for varname in inspect.currentframe().f_code.co_varnames:
        if varname == 'obj':
            continue
        else:
            setattr(obj, varname, locals()[varname])

>>> autoinit(os, 5, 10)
>>> os.x
5
>>> os.y
10

So far, so good!  We now have a function that can use the parameter names in setting attributes.  However, if this function is going to work, then it’s not going to exist on its own. Rather, we want to invoke “autoinit” from within our __init__ method.  This means that we need autoinit to set attributes not based on its own parameters, but rather based on its caller’s parameters.  The inspect.currentframe method returns the current stack frame, but we want the caller’s stack frame.

Fortunately, the implementers of inspect.currentframe thought of this, and provide an easy and elegant solution: If we invoke inspect.currentframe with a numeric parameter, Python will jump back that number of stack frames.  Thus, inspect.currentframe() returns the current stack frame, inspect.currentframe(1) returns the caller’s stack frame, and inspect.currentframe(2) inspect the caller’s caller’s stack frame.

By invoking inspect.currentframe(1) from within __init__, we can get the instance onto which we want to add the attributes, as well as the attribute names and values themselves. For example:

import inspect
def autoinit():
    frame = inspect.currentframe(1)
    params = frame.f_locals
    self = params['self']
    paramnames = frame.f_code.co_varnames[1:] # ignore self
    for name in paramnames:
        setattr(self, name, params[name])

class Foo(object):
    def __init__(self, x, y):
        autoinit()

>>> f = Foo(100, 'abc')
>>> f.x
100
>>> f.y
'abc'
>>> g = Foo(200, 'ghi')
>>> g.x
200
>>> g.y
'ghi'

Hey, that’s looking pretty good!  However, there is still one problem: Python doesn’t see a difference between parameters and local variables.  This means that if we create a local variable within __init__, autoinit will get confused:

class Bar(object):
    def __init__(self, x, y):
        autoinit()
        z = 999
>>> b = Bar(100, 'xyz')
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
 File "<stdin>", line 3, in __init__
 File "<stdin>", line 7, in autoinit
KeyError: 'z'

As you can see, autoinit tries to find the value of our ‘z’ local variable — but because the variable has not been set, it isn’t in the locals() dictionary.  But the bigger problem is that autoinit is trying to look for z at all — as a local variable, rather than a parameter, we don’t want it to be set as an attribute on self!

The solution is to use the co_argcount attribute to our code object, which says how many arguments our function takes. For example:

def foo(x):
    y = 100
    return inspect.currentframe()

>>> s = foo(5)
>>> print(s.f_code.co_argcount)
1
>>> print(s.f_code.co_varnames)
('x', 'y')

We can improve our implementation of autoinit by only taking the first co_argcount elements of co_varnames. So far as I can tell (and I don’t know if this is official, or just a convenient accident), the arguments always come first in co_varnames.  Our final version of autoinit thus looks like this:

def autoinit():
    frame = inspect.currentframe(1)
    params = frame.f_locals
    nparams = frame.f_code.co_argcount
    self = params['self']
    paramnames = frame.f_code.co_varnames[1:nparams]
    for name in paramnames:
        setattr(self, name, params[name]) 

Sure enough, if we try it:

class Foo(object):
    def __init__(self, x, y):
        autoinit()
        z = 100
>>> f = Foo(10, 20)
>>> print f.x
10
>>> print f.y
20

Success!  Of course, this implementation still doesn’t handle *args or **kwargs.  And as I wrote above, it’s very much not in the spirit of Python to have such magic happening behind the scenes.  Yet, I’ve found this to be an interesting way to discover and play with functions, the “inspect” module, and how arguments are handled.

If you have used Python’s introspection in a clever or interesting way, please let me know; I would love to hear from you!  And of course, if this is the sort of thing that interests you, be sure to check out my three-day, online Python course, starting on January 14th, 2014.

Python scoping e-mail course

I had so much fun writing the previous blog post about Python scoping that I decided to expand it into a free e-mail course.  Each day (for five days), you’ll receive another lesson about how scopes work in Python, and why this is important for you to know as a Python developer.

So if you’ve ever been unclear on the “LEGB rule,” wanted to know when and how to use the “global keyword,” or even how you can nearly break your Python implementation through scope abuse, this e-mail course should help.

Please e-mail me if you have questions or comments about this e-mail course!  I’ve had so much fun putting this one together that I’m very likely to create additional ones, so your suggestions for future topics are extremely welcome.

 

Fun with Python scoping rules

Let’s say I want to try something on a list in Python.  While I usually like to call my test list objects “mylist”, I sometimes forget, and create a variable named “list”:

list = ['a', 'b', 'c']

If you’re like me, then you might not immediately notice that you’ve just defined a variable whose name is the same as a built-in type.  Other languages might have defined “list” as a reserved word, such that you cannot define it.  (Just try creating a variable named “if”, and you’ll see what I mean.)

But Python won’t stop you.  This means that now, instead of type(list) returning “type” (i.e., indicating that list is a data type), it’ll say:

>>> type(list)
<type 'list'>

If you’re new to Python, and think that it’s normal for the return type of “list” to be “list”, let’s get a bit bolder:

>>> list = 'abc'
>>> type(list)
<type 'str'>

(If you’re using Python 3, then you’ll see “<class ‘str’>”, and not “<type ‘str’>”.   But there’s really no difference.)

Now things are getting downright weird.  Of course, I can fix the situation:

>>> del(list)
>>> type(list)
<type 'type'>

What’s going on here?  How was I able to turn lists into strings?  And how did deleting “list” suddenly restore things?

The answer is: Python’s scoping rules.  They are fairly simple to understand, and very consistent (as you would expect from Python), but have implications that can cause all sorts of weirdness if you’re not sure of what to expect.

The Python scoping rules are Local, Enclosing Function, Global, and Builtins, often abbreviated as “LEGB.”  This means that when Python encounters an identifier (variable or function) name, it will look for the name as follows:

  1. If you’re inside of a function, it’ll first look in that function,
  2. If you have defined a function within a function (and beginners really shouldn’t be doing this), then Python will look in the enclosing function,
  3. Python then looks at the “global” level, which is another way of saying in the current file, and
  4. As a last resort, Python looks in the __builtins__ module, the namespace in which Python’s standard and built-in types are located.

So if I have a one-line Python program:

print(x)

there are no functions, and thus no L or E to consider.  Python will look in G, the global namespace, meaning the current file.  If there is an x defined there, then great; that’s what will be printed.  If there is no x defined in the current file, then Python will look in __builtins__, which doesn’t have an x, and we’ll get a NameError exception — meaning, Python doesn’t know what name you’re talking about.

So far, so good, right?  Well, now consider what happens when we define a new variable, by assigning it a value.  If you define the variable inside of a function, then it’s in the “local” scope.  If you define the variable at the top level of a file, then it’s in the “global” scope.  And if you define it in the __builtins__ module (and you really shouldn’t be doing that), then it’ll be in the “builtins” scope.

When we defined “list”  (e.g., “list = ‘abc’”), we were defining a new variable in the global scope. We didn’t replace or remove the builtin “list” at all!  Indeed, the builtin “list” is still available if we use its full name, __builtins__.list:

>>> list = 'abc'
>>> type(list)
<type 'str'>
>>> type(__builtins__.list)
<type 'type'>

The problem, then, isn’t that we have replaced the built-in “list”, but rather that we have masked it.  Once we have defined “list” in the global scope, all naked references to “list” in that file — as well as in functions defined within that file — will see our global variable “list”, rather than __builtins__.list.

How, then, did it help for me to delete “list”?  Because del(list) doesn’t delete __builtins__.list. (You can do that, by the way, but that’s for another blog post.)  Rather, del(list) in our case deletes “list” from the global scope.  When we then ask Python for type(list), it looks in L, E, and G, and doesn’t find anything.  It thus goes to __builtins__, finds “list”, and returns us the type of “list”, which is once again a “type” or “class”.  Whew!

If you enjoyed this, then you might like my three-day, online Python class, which includes such tidbits.  The course will begin on January 14th (read more about it at masterpython.com).  You can also subscribe to my free, exclusive technology newsletter.