Yes, you can master regular expressions!

Announcing: My new book, “Practice Makes Regexp,” with 50 exercises meant to help you learn and master regular expressions. With explanations and code in Python, Ruby, JavaScript, and PostgreSQL.

I spend most of my time nowadays going to high-tech companies and training programmers in new languages and techniques. Actually, many of the things I teach them aren’t really new; rather, they’re new to the participants in my training. Python has been around for 25 years, but for my students, it’s new, and even a bit exciting.

I tell participants that my job is to add tools to their programming toolbox, so that if they encounter a new problem, they’ll have new and more appropriate or elegant ways to attack and solve it. Moreover, I tell them, once you are intimately familiar with a tool or technique, you’ll suddenly discover opportunities to use it.
Earlier this week, I was speaking with one of my consulting clients, who was worried that some potentially sensitive information had been stored in their Web application’s logfiles — and they weren’t sure if they had a good way to search through the logs.

 

I suggested the first solution that came to mind: Regular expressions.

Regular expressions are a lifesaver for anyone who works with text.  We can use them to search for patterns in files, in network data, and in databases. We can use them to search and replace.  To handle protocols that have changed ever so slightly from version to version. To handle human input, which is always messier than what we get from other computers.

Regular expressions are one of the most critical tools I have in my programming toolbox.  I use them at least a few times each day, and sometimes even dozens of times in a given day.

So, why don’t all developers know and use regular expressions? Quite simply, because the learning curve is so steep. Regexps, as they’re also known, are terse and cryptic. Changing one character can have a profound impact on what text a regexp matches, as well as its performance. Knowing which character to insert where, and how to build up your regexps, is a skill that takes time to learn and hone.

Many developers say, “If I have a problem that involves regular expressions, I’ll just go to Stack Overflow, where my problem has likely been addressed already.” And in many cases, they’re right.

But by that logic, I shouldn’t learn any French before I go to France, because I can always use a phrasebook.  Sure, I could work that way — but it’s far less efficient, and I’ll miss many opportunities that would come my way if I knew French.

Moreover, relying on Stack Overflow means that you never get a full picture of what you can really do with regular expressions. You get specific answers, but you don’t have a fully formed mental model of what they are and how they work.

But wait, it gets worse: If you’re under the gun, trying to get something done for your manager or a big client, you can’t spend time searching through Stack Overflow. You need to bring your best game to the table, demonstrating fluency in regular expressions.  Without that fluency, you’ll take longer to solve the problem — and possibly, not manage to solve it at all.

Believe me, I understand — my first attempt at learning regular expressions was a complete failure. I read about them in the Emacs manual, and thought to myself, “What could this seemingly random collection of characters really do for me?”  I ignored them for a few more years, until I started to program in Perl — a language that more or less expected you to use regexps.

So I spent some time learning regexp syntax.  The more I used them,  the more opportunities I found to use them.  And the more I found that they made my life easier, better, and more convenient.  I was able to solve problems that others couldn’t — or even if they could, they took much longer than I did.  Suddenly, processing text was a breeze.

I was so excited by what I had learned that when I started to teach advanced programming courses, I added regexps to the syllabus.  I figured that I could figure out a way to make regexps understandable in an hour or two.

But boy, was I wrong: If there’s something that’s hard for programmers to learn, it’s regular expressions.  I’ve thus created a two-day course for people who want to learn regular expressions.  I not only introduce the syntax, but I have them practice, practice, and practice some more.  I give them situations and tasks, and their job is to come up with a regexp that will solve the problem I’ve given them.  We discuss different solutions, and the way that different languages might go about solving the problem.

After lots of practice, my students not only know regexp syntax — they know when to use it, and how to use it.  They’re more efficient and valuable employees. They become the person to whom people can turn with tricky text-processing problems.  And when the boss is pressuring them for a

ImageAnd so, I’m delighted to announce the launch of my second ebook, “Practice Makes Regexp.”  This book contains 50 tasks for you to accomplish using regular expressions.  Once you have solved the problem, I present the solution, walking you through the general approach that we would use in regexps, and then with greater depth (and code) to solve the problem in Python, Ruby, JavaScript, and PostgreSQL.  My assumption in the book is that you have already learned regexps elsewhere, but that you’re not quite sure when to use them, how to apply them, and when each metacharacter is most appropriate.

After you go through all 50 exercises, I’m sure that you’ll be a master of regular expressions.  It’ll be tough going, but the point is to sweat a bit working on the exercises, so that you can worry a lot less when you’re at work. I call this “controlled frustration” — better to get frustrated working on exercises, than when the boss is demanding that you get something done right away.

Right now, the book is more than 150 pages long, with four complete chapters (including 17 exercises).  Within two weeks, the remaining 33 exercises will be done.  And then I’ll start work on 50 screencasts, one for each of the exercises, in which I walk you through solutions in each of Python, Ruby, JavaScript, and PostgreSQL.  If my previous ebook is any guide, there will be about 5 hours (!) of screencasts when I’m all done.

If you have always shied away from learning regular expressions, or want to harness their power, Practice Makes Regexp is what you have been looking for.  It’s not a tutorial, but it will help you to understand and internalize regexps, helping you to master a technology that frustrates many people.

To celebrate this launch, I’m offering a discount of 10%.  Just use the “regexplaunch” offer code, and take 10% off of any of the packages — the book, the developer package (which includes the solutions in separate program files, as well as the 300+ slides from the two-day regexp course I give at Fortune 100 companies), or the consultant package (which includes the screencasts, as well as what’s in the developer package).

I’m very excited by this book.  I think that it’ll really help a lot of people to understand and use regular expressions.  And I hope that you’ll find it makes you a more valuable programmer, with an especially useful tool in your toolbox.

All 50 “Practice Makes Python” screencasts are complete!

I’m delighted to announce that I’ve completed a screencast for every single one of the 50 exercises in my ebook, “Practice Makes Python.”  This is more than 300 minutes (5 hours!) of Python instruction, helping you to become a more expert Python programmer.

Each screencast consists of me solving one of the exercises in real-time, describing what I’m doing and what I’m doing it.   They range in length from 4 to 10 minutes.  The idea is that you’ll do the exercise, and then watch my video to compare your answer (and approach) with mine.

If you enjoy my Webinars or in-person courses, then I think you’ll also enjoy these videos.

The screencasts, available with the two higher-tier “Practice Makes Python” packages,  can be streamed in HD video quality, or can be downloaded (DRM-free) to your computer for more convenient viewing.

To celebrate finally finishing these videos, I’m offering the two higher-end packages at 20% off for the coming week, until February 18th. Just use the offer code “videodone” with either the “consultant” or “developer” package, and enjoy a huge amount of Python video.

You can explore these packages at the “Practice Makes Python” Web site.

Not interested in my book, but still want to improve your Python skills?  You can always take one of my two free e-mail courses, on Python variable scoping and working with files. Those are and will remain free forever. And of course, there’s my free Webinar on Python and data science next week.

Free Webinar: Pandas and Matplotlib

It’s time for another free hour-long Webinar! This time, I’ll be talking about the increasingly popular tools for data science in Python, namely Pandas and Matplotlib. How can you read data into Pandas, manipulate it, and then plot it? I’ll show you a large number of examples and use cases, and we’ll also have lots of time for Q&A. Previous Webinars have been lots of fun, and I expect that this one will be, too!

Register (for free) to participate here:

https://www.eventbrite.com/e/analzying-and-viewing-data-with-pandas-and-matplotlib-tickets-21198157259

If you aren’t sure whether you’ll be able to make it, you can still sign up; I’ll be sending information, and a URL with the recording afterwards, soon after the Webinar concludes.

I look forward to seeing you there; if you have any questions, please feel free to contact me at reuven@lerner.co.il or on Twitter as @reuvenmlerner.

Reminder: Free Webinar on data science in Python

There’s still time to register for my free, one-hour Webinar on data science in Python, which will be tomorrow (Tuesday).  There’s clearly too much material for me to give just one Webinar, so this will be the first in a series that I’ll be offering over the coming months.  But if you’re interested in hearing how Python fits into the world of data science, or how you can use free, open-source tools to do lots of great analysis work, then I invite you to join me for what should be a fun time:

https://www.eventbrite.com/e/data-analysis-with-python-tickets-19543502141

There will be plenty of live-coding demos, bad jokes, and chances for you to ask questions. And it should be lots of fun, besides!

Free Webinar: Data science with Python on December 8th

It’s time for me to do another free one-hour Webinar, this time about data science with Python. It’ll be on December 8th, at 9 p.m. GMT.

Data science is all the rage, and rightly so — and Python is one of the best-known and best-equipped languages in which to do it.  In this Webinar, I’ll review some of the most popular packages used for analysis, including NumPy, SciPy, Pandas, and matplotlib, and will show how they can be used to answer questions that we have about our data.

As always, I hope that there will be lots of questions — and if we’re lucky, I’ll be able to provide answers, too!  Please come prepared for a highly interactive and fun event.  I’ll e-mail all registered participants about an hour before the Webinar with links for participating.

You can register at Eventbrite: https://www.eventbrite.com/e/data-analysis-with-python-tickets-19543502141

I look forward to seeing you there!  If you have any questions, please let me know via e-mail at reuven@lerner.co.il or on Twitter as @reuvenmlerner.

Python’s objects and classes — a visual guide

Python developers love to say that “everything is an object.” And indeed, when I teach Python classes, I say this several times, and many people nod in agreement, assuming that I’m merely repeating something they’ve heard before.  After all, people often say that everything in Java is an object (except for the things that aren’t), and that everything in .NET is an object.

But when we say that everything in Python is an object, we really mean everything, including — much to the surprise of my students — classes. This makes enormous sense, and it makes the entire object system easier to understand. And yet, it is still hard to put things in perspective.

In this blog post, I want to walk through some of the connections that we have among objects in Python, in the hopes that it’ll help to cement some of the ideas that stem from this “everything is an object” idea. It’ll also demonstrate some of the fun that happens when you’re creating an object hierarchy, and how things can get a bit weird.

Let’s start with a simple class (MyClass), and a simple instance of that class (m). In Python, we would write:

class MyClass(object):
    pass

m = MyClass()

In Python 3, we don’t need to explicitly say that MyClass inherits from object, since that’s true for all classes. But in Python 2, we have to inherit from object; if we don’t, then we get old-style classes, which we really don’t want.

Let’s see how this looks visually, with the arrow indicating that m is an instance of MyClass:

Python objectsSo far, that’s not very exciting. But let’s remember that everything in Python is an object. Thus, it’s true that m is an instance of MyClass; we can learn this by using the type function:

>>> type(m)
__main__.MyClass

What happens if we ask MyClass about its type?

>>> type(MyClass)
type

Yes, MyClass is an instance of type — just as str, int, bool, and other Python classes are instances of type. Our diagram has just gotten a bit more complex:

Python objects 2

In the above diagram, we see that m is an instance of MyClass, and MyClass is an instance of type.

One main difference between regular objects and classes is that classes have a __bases__ attribute, a tuple, which indicates from which other class(es) this class inherits. MyClass, like all classes, should really have two pointers in our diagram — one representing its type, and another representing from which class (object) it inherits:

Python objects 3Many of the people to whom I teach Python are confused by the distinction between type and object, and what roles they play in the object’s life. Consider this:

  • Because MyClass is an instance of type, type.__init__ determines what happens to our class when it is created.
  • Because MyClass inherits from object, invoking a method on m will result in first looking for that method on MyClass. If the method doesn’t exist on MyClass, then Python will look on object.

All of this is well and good, but let’s take it a bit further: We know that MyClass is an instance of type. But this means that type itself is a class, right? What is the type of this type class?

>>> type(type)
type

Yes, in one of my favorite parts of Python, the type of type is type. In other words, type is an instance of itself. Pretty cool, eh? Let’s see how that fits into our diagram:

Python objects 4

If type is a class, then we know it must have two pointers in our diagram — one pointing to its class (type, aka itself), but the one to the class from which it inherits. What does type inherit from?

>>> type.__bases__
(object,)

Let’s thus update our diagram, to show that type inherits from object. This makes sense, since if we invoke str(MyClass), we can rely on the inherited  implementation of object.__str__, without having to create a separate type.__str__.  And indeed, it would seem that this is what happens:

>>> type.__str__ is object.__str__
True

Let’s now update our diagram to indicate that type inherits from object:

Python objects 5

Finally, let’s not neglect our object class. As an object, it too must have a type. And as a class, we know that its type is type. Let’s add that to our diagram:

Python objects 6Remember that object is at the top of our inheritance hierarchy. This is represented in Python by an empty tuple:

>>> object.__bases__
()

We can represent this in our diagram in the following way:

Python objects 7Finally, let’s see what happens when we add a new class to this hierarchy, subclassing from MyClass. MySubClass inherits from MyClass, but is still an instance of type:

Python objects 8

If you’re an experienced Python developer, then the above may well be second nature for you. But if you’re new to the language, and particularly to the ways in which the various objects and classes interact, then I hope this has provided you with some additional clarity. Please let me know if there are additional aspects that you find confusing, and I’ll try to clarify them in future blog posts.

If you liked this explanation, then you’ll likely also enjoy my ebook, “Practice Makes Python,” with 50 exercises meant to improve your Python fluency.

Registration is open for my October Webinars (about regexps and technical training)

September has been busy with work and holidays, but I’m gearing up for an exciting and busy October. Among other things, I’m giving two (free) Webinars in that month, and you can already register for them:

  • Intermediate Regular expressions: In my previous Webinar about regular expressions, I covered the basics.  In this one, we’ll go further, spending a great deal of time talking about groups, backreferences, and some other topics that tend to confuse people.  I’ll be using Python to demonstrate regexps, but this Webinar isn’t only aimed at Python developers. Registration is free; sign up here.
  • Technical training: I’m starting to spend more and more time helping people to become technical trainers, teaching programming in high-tech companies. (Indeed, I’m in the process of starting my coaching program for people interested in improving their training skills.) In this Webinar, which I expect will be the first of many, I’ll give an overview of the technical-training landscape, how it works, and why you should seriously consider providing training services. I’ll briefly review pedagogical, logistical, and business considerations for the aspiring technical trainer. Registration for this training is free; you can sign up here.

Understanding nested list comprehensions in Python

In my last blog post, I discussed list comprehensions, and how to think about them. Several people suggested (via e-mail, and in comments on the blog) that I should write a follow-up posting about nested list comprehensions.

I must admit that nested list comprehensions are something that I’ve shied away from for years. Every time I’ve tried to understand them, let alone teach them, I’ve found myself stumbling for words, without being clear about what was happening, what the syntax is, or where I would want to use them. I managed to use them on a few occasions, but only after a great deal of trial and error, and without really understanding what I was doing.

Fortunately, the requests that I received, asking how to work with such nested list comprehensions, forced me to get over my worries. I’ve figured out what’s going on, and even think that I understand what my problem was with understanding them before.

Get the bonus content: Nested list comprehensions

The key thing to remember is that in a list comprehension, we’re dealing with an iterable. So when I say:

[ len(line) 
for line in open('/etc/passwd') ]

I’m saying that I want to iterate over the file object we got from opening /etc/passwd. There will be one element in the output list for each element in the input iterable — aka, every line in the file.

That’s great if I want my list comprehension to return something based on each line of /etc/passwd. But each line of /etc/passwd is a string, and thus also iterable. Maybe I want to return something not based on the lines of the file, but on the characters of each line.

Were I to use a “for” loop to process the file, I would use a nested loop — i.e., one loop inside of the other, with the outer loop iterating over lines and the inner loop iterating over consonants. It turns out that we can use a nested list comprehension, too. Here’s a simple example of a nested list comprehension:

[(x,y) for x in range(5) for y in range(5)]

If your reaction to this is, “What in the blazes does that mean?!?” then you’re not alone. Until just recently, that’s what I thought, too.

However: If we rewrite the above nested list comprehension using my preferred (i.e., multi-line) list-comprehension style, I think that things become a bit clearer:

 [(x,y)  
 for x in range(5)  
 for y in range(5)]

Let’s take this apart:

  • Our output expression is the tuple (x,y). That is, this list comprehension will produce a list of two-element tuples.
  • We first run over the source range(5), giving x the values 0 through 4.
  • For each value in x, we run through the source range(5), giving y the values 0 through 4.
  • The number of values in the output depends on the number of runs of  the final (second) “for” line.
  • The output, not surprisingly, will be all of the two-element tuples from (0,0) to (4,4).

Now, let’s mix things up by changing them a bit:

 [(x,y)  
  for x in range(5)  
  for y in range(x+1)]

Notice that now, the maximum value of y will vary according to the value of x. So we’ll get from (0,0) to (4,4), but we won’t see such things as (2,4) because y will never be larger than x.

Again, it’s important to understand several things here:

  • Our “for y” loop will execute once for each iteration over x.
  • In our “for y” loop, we have access to the variable x.
  • In our “for x” loop, we don’t have access to y (unless you consider the last value of y to be useful, but you really shouldn’t).
  • Our (x,y) tuple is output once for each iteration of the *final* loop, at the bottom.

Here’s another example: Assume that we have a few friends over, and that we have decided to play several games of Scrabble. Being Python programmers, we have stored our scores in a dictionary:

{'Reuven':[300, 250, 350, 400], 
 'Atara':[200, 300, 450, 150], 
 'Shikma':[250, 380, 420, 120], 
 'Amotz':[100, 120, 150, 180] }

I want to know each player’s average score, so I write a little function:

def average(scores):  
    return sum(scores) / len(scores)

If we want to find out each individual’s average score, we can use our function and a standard comprehension — in this case, a dict comprehension, to preserve the names:

 >>> { name : average(score)  
       for name, score in scores.items() }

{'Amotz': 137, 'Atara': 275, 'Reuven': 325, 'Shikma': 292}

But what if I want to get the average score, across all of the players? In such a case, I will need to grab each of the scores from inside of the inner lists. To do that, I can use a nested list comprehension:

>>> average([ one_score  
              for one_player_scores in scores.values()  
              for one_score in one_player_scores ])

257

What if I’m only interested (for whatever reason) in including scores that were above 200? As with all list comprehensions, I can use the “if” clause to weed out values that I don’t want. That condition can use any and all of the values that I have picked out of the various “for” lines:

>>> [ one_score      
      for one_player_scores in scores.values()     
      for one_score in one_player_scores
      if one_score > 200]

[300, 250, 350, 400, 300, 450, 250, 380, 420]

If I want to put these above-200 scores into a CSV file of some sort, I could do the following:

>>> ','.join([ str(one_score)  
               for one_player_scores in scores.values() 
               for one_score in one_player_scores  
               if one_score > 200])

'300,250,350,400,300,450,250,380,420'

Here’s one final example that I hope will drive these points home: Let’s assume that I have information about a hotel. The hotel has stored its information in a Python list. The list contains lists (representing rooms), and each sublist contains one or more dictionaries (representing people). Here’s our data structure:

rooms = [[{'age': 14, 'hobby': 'horses', 'name': 'A'},  
          {'age': 12, 'hobby': 'piano', 'name': 'B'},  
          {'age': 9, 'hobby': 'chess', 'name': 'C'}],  
         [{'age': 15, 'hobby': 'programming', 'name': 'D'}, 
          {'age': 17, 'hobby': 'driving', 'name': 'E'}],  
         [{'age': 45, 'hobby': 'writing', 'name': 'F'},  
          {'age': 43, 'hobby': 'chess', 'name': 'G'}]]

What are the names of the people staying at our hotel?

 >>> [ person['name']      
       for room in rooms
       for person in room ]

['A', 'B', 'C', 'D', 'E', 'F', 'G']

How about the names of people staying in our hotel who enjoy chess?

>>> [ person['name']  
      for room in rooms  
      for person in room  
      if person['hobby'] == 'chess' ]

['C', 'G']

Basically, every “for” line flattens the items over which you’re iterating by one more level, gives you access to that level in both the output expression (i.e., first line) and in the condition (i.e., optional final line).

I hope that this helps you to understand nested list comprehensions. If it did, please let me know! (And if it didn’t, please let me know that, as well!)

Get the bonus content: Nested list comprehensions

Want to understand Python’s comprehensions? Think in Excel or SQL.

Comprehensions are among the most useful constructs in Python. They merge the old, trusty “map” and “filter” functions into a single piece of compact, elegant syntax, allowing us to expression complex ideas in a minimum of code. Comprehensions are one of the most important tools in a Pythonista’s toolbox.

And yet, I have found that a very large number of Python programmers, including some experienced developers, are not completely comfortable with comprehensions. There are two reasons for this: First, it’s not obvious when to use them, and what sorts of problems they solve. The second problem, which is at least as important, is that the syntax is hard for people to remember and understand.

I’ve started to use a new explanation and introduction to comprehensions in my Python classes, and have found that it helps to lower the learning curve to some degree. In this post, I’m publicizing this explanation, in the hopes that it’ll help Python developers to understand when, where, and how to use comprehensions.

Let’s take a simple problem: I want to take a list of five integers, and get a list of their squares. If you give this problem to a new (or even intermediate) Python programmer, the answer would look something like this:

numbers = range(5)
output = [ ]
for number in numbers:
    output.append(number * number)
print(output)

Now, the thing is that this does work. (In my courses, I often use the phrase, “Unfortunately, this works.”) Often, when I talk about comprehensions, I talk about functional programming, the idea of immutable data structures, the idea that we don’t want to change things, and the benefits of thinking in terms of mapreduce.

But let’s ignore all of that, and ask a simpler question: If you were to give this problem to your accountant, how would they solve the problem?

Almost certainly, an accountant would fire up Excel, and put the numbers in a column:

A
-
0
1
2
3
4

Let’s assume that the above numbers are in the spreadsheet’s column A. The Excel user would, given this task, then tell Excel that column B should be calculated as A*A. And it would be done:

A  B
-  -
0  0
1  1
2  4
3  9
4  16

You could argue that the difference here is that Excel has a GUI, and Python doesn’t. But that’s missing the point. The real difference is that our accountant told Excel how to transform the first column into the second column, whereas our Python developer wrote a program that describe how to carry out that transformation.

We can think about this in a different way, too: Rather than solving the problem serially, as in the above for loop, the accountant is thinking in a parallel manner, applying a single expression to a large data set. The Excel user doesn’t care, or even know, the order in which the numbers are handed to the expression. The important thing is that the expression is applied once to each of the numbers, and that the final result appears in the correct order.

We might laugh at Excel, and dismiss its users as technical neophytes. And certainly, many users of Excel would deny that they possess serious programming chops. But this sort of thinking, which is so fundamental and natural to Excel users, is alien to many programmers. Which is a shame, because it allows us to express a very large number of ideas in a simple way.

To summarize this approach:

  • Think of your input as an iterable source of data
  • Think of what operation you want to apply to each element of that source
  • Get a new sequence out

That’s what the traditional “map” function does. Python does have a “map” function, but today, we typically use list comprehensions instead.

Let’s try to make this a bit more concrete, using the example that I used above: Let’s say that we have a list of five numbers, and we want to turn that list into a list of its squares. The list-comprehension syntax looks as follows:

[number * number for number in range(5) ]

Yikes. No wonder people are scared off by this syntax.  Let’s take the above syntax apart:

  • First of all, we’re going to get a list back. (It’s called a “list comprehension” for a reason.) That’s because of the square brackets, which are mandatory, and which tell Python what sort of object to create.
  • The data source will be “range(5),” which returns a list.
  • Each element in the data source will be assigned, in turn, to the iteration variable “number.”
  • We’ll invoke the operation “number * number” on each element of the data source.

In other words, we’re creating a new list, the elements of which are the result of applying our expression to each element of the source. This sounds suspiciously like what our accountant did above, using Excel: We’re telling Python what we want, and how to transform our source to that result. But how are things done internally? How is the list created? We neither know nor care.

List-comprehension syntax can be daunting for people to understand, in part because the order of the operations seems unusual. I’ve found that it can help to rewrite list comprehensions in the following way:

[number * number
 for number in range(5) ]

Yes, that’s right — I now spread list comprehensions across two lines; the first describes the operation I want to invoke, and the second line describes the data source. If this still seems unfamiliar, let’s try to bring it into a context with which you might have some experience:

[number * number           # SELECT
 for number in range(5) ]  # FROM

While they’re not directly equivalent, there are a fair number of similarities between a SELECT query in SQL, the placement of its SELECT expression and FROM clause, and our list comprehension.  The FROM clause in an SQL query describes our data source, which is typically going to be a table, but can also be a view or even the result of a function call. And the initial part of the SELECT is often the name of a column, but  can include function calls and operators.

On the one hand, the SELECT-FROM combination seems almost too simple to mention, in that you’re just retrieving a selected set of values from a data source.  On the other hand, such queries form the backbone of the database industry. In the same way, such functionality forms the backbone of many Python programs, iterating over a data structure, and plucking out part of it, transforming that part, and then returning a new list.

One of my favorite examples (and an exercise in my ebook, “Practice Makes Python“) is to take the /etc/passwd file used in Unix, and get the usernames contained within that file. /etc/passwd consists of one record per line, and the fields are separated by colons. Here are several lines from the /etc/passwd on my computer:

nobody:*:-2:-2::0:0:Unprivileged User:/var/empty:/usr/bin/false
root:*:0:0::0:0:System Administrator:/var/root:/bin/sh
daemon:*:1:1::0:0:System Services:/var/root:/usr/bin/false
_uucp:*:4:4::0:0:Unix to Unix Copy Protocol:/var/spool/uucp:/usr/sbin/uucico

We might normally think of a file as a collection of bytes, to which we give semantic meaning when we read it. But in Python, we’re encouraged to see a file as an ordered, iterable collection of lines of text. True, I can read from a file based on bytes, but it’s so common to want to read files by line that the language provides several constructs to do so.

We know that we can iterate over the lines of a file:

for line in open('/etc/passwd'):
    print(line)

This demonstrates that a file is iterable, which means that it can serve as a data source for a list comprehension. This means that the above code can be rewritten as:

[line
 for line in open('/etc/passwd')]

Again, the first line in our list comprehension represents the expression we want to apply to every element of our data source. In this case, the expression is just the line.  If we want to get the username from each of  these lines, we just need to apply the “split” method on the string, returning a list — and then retrieve index 0 from the resulting list.  For example:

[line.split(":")[0]
 for line in open('/etc/passwd')]

Again, we can think of it in terms of an SQL query:

SELECT username
FROM users

But of course, “username” in the above is a column name.  A more equivalent query to my list comprehension would be a “Users” table with an “info” column, queried as follows:

SELECT split_part(info, ':', 1)
FROM users;

Note that in this case, I’m using the built-in PostgreSQLsplit_part” operator to perform the equivalent operation to the str.split method in Python.

Remember that in the case of my SQL query, the result of a query always looks and acts like a table. The number and types of columns returned will depend on the number and types of expressions that I have in the SELECT  statement.  But the result set will have one or more columns, and zero or more rows.

In the same way, the result of a list comprehension is always going to be a list.  You can have whatever expression you want inside of the list comprehension; the expression represents one item in a list, not the list itself.

For example, let’s assume that I want to turn the usernames in /etc/passwd into a list of dictionaries. This doesn’t require a dictionary comprehension, which creates a single dictionary.  Rather, it requires a list  comprehension, in which the expression creates a dictionary.  Here’s a simple-minded such list comprehension:

[ {'name':line.split(":")[0]}
   for line in open('/etc/passwd')]

The above will work, in that it creates a list of dictionaries. And each dictionary has a single key-value pair.  But it seems a bit silly to do the above.  Rather, I’d probably want to have a dictionary containing the username and the numeric user ID, which is at index 2. I can then write:

[ {'name':line.split(":")[0], 'id':line.split(":")[2]}
for line in open('/etc/passwd')]

Again, we can think about this in terms of Excel, or even in terms of SQL: My query now produces a single column of results, but each column contains a text string. Or we can even say that the query produces two columns of results, which is not at all unusual in the world of SQL.

Let’s ignore the efficiency (or lack thereof) of invoking str.split twice in one comprehension: When I run this code on my Mac, it results in an exception, claiming that an index is out of range.

The reason is simple: I split each line into a list. But if there’s a line that doesn’t contain any : characters, it’ll be turned into a single-element list. I thus need to weed out any lines that won’t conform. Specifically, on my Mac at least, I need to remove any lines in /etc/passwd that are comments, meaning that they start with the ‘#’ character.

In the world of list comprehensions, I say the following:

[ {'name':line.split(":")[0], 'id':line.split(":")[2]}
for line in open('/etc/passwd')
if not line.startswith("#")]

Let’s extend our earlier SQL analogy further, adding the equivalent SQL syntax in comments after our Python code:

[ {'name':line.split(":")[0], 'id':line.split(":")[2]}    # SELECT
for line in open('/etc/passwd')                           # FROM
if not line.startswith("#")]                              # WHERE

Of course, when the first line of our comprehension becomes this long, it’s often a good idea to use a function, instead. And since the first line can be any legitimate Python expression, a function is often a good idea:

def get_user_info(line):
    name, passwd, id, rest = line.split(":", 3)   # max 4 fields
    return {'name':name, 'id':id}

[ get_user_info(line)             # SELECT
for line in open('/etc/passwd')   # FROM
if not line.startswith("#")]      # WHERE

A list comprehension thus gives you power similar to an SQL SELECT query — except that you’re not querying data in a table, but rather any object that conforms to Python’s iteration protocol, which includes a very  large number of built-in and custom-made objects.

Now, when would you want to use a list comprehension? And how does it differ from a for loop?

Using a list comprehension is appropriate whenever you want to transform data. That is, you have an iterable data source, and you want to create a new list whose elements are based on those of the data source. For  example, let’s assume that (for some reason) I want to find out how many times each character is used in /etc/passwd.  I can thus do the following, using collections.Counter:

from collections import Counter
counts = [Counter(line)
          for line in open('/etc/passwd')
          if not line.startswith("#")]

We know that “counts” is a list, because I used a list comprehension to create it. It is a list containing many Counter objects, one for each non-comment line in /etc/passwd. What if I want to find out what the most  popular character is in each line? I can modify my expression, asking the Counter object for the most common character:

counts = [Counter(line).most_common(1)
          for line in open('/etc/passwd')
          if not line.startswith("#")]

I can extend my expression even more, to get the most popular character from each line (inside of a two-element tuple in a one-element list):

counts = [Counter(line).most_common(1)[0][0]
          for line in open('/etc/passwd')
          if not line.startswith("#")]

And now I can find out how many times each most-popular character appears:

Counter([Counter(line).most_common(1)[0][0]
          for line in open('/etc/passwd')
          if not line.startswith("#")])

On my computer, the answer is:

Counter({':': 71, 'e': 4, 's': 1})

Meaning that in 71 non-comment lines, “:” is the most common, but in 4 lines it’s “e”, and in one line it’s “s”.  Now, could I have done this with a for loop?  Yes, of course — but because I’m dealing with iterables, and  because I’m using objects that work with such iterables, I can chain them together to get an answer in a way that doesn’t require me to tell Python how to do its job. I’m doing things like our accountant did, back at the  start of this article — I’m saying what I want, and letting Python do the hard work of dealing with this for me.

When would I use a for loop, then? The distinction is between whether you want to get a list back, and whether you want to execute a command a number of times.  If you want to build a list, and if it’s built on an iterable that already exists, then I’d say a list comprehension is almost certainly going the be the best bet.  But if you want to execute something a number of times without creating a list, then a comprehension is the a bad way to do it; you should use a “for” loop, instead.

It’s true that list comprehensions are faster than for loops. But most of the time, for loops are used for different things than list comprehensions. “for” loops shouldn’t be used when you want to turn one iterable structure into another; that’s for comprehensions. And you shouldn’t execute something (e.g., print) many times via a list comprehension, even if you can do so via a called function.  I’ve found that the dividing line between when to use a “for” loop, and when to use a comprehension, is clearly delineated in the minds of experienced Python developers, but very hazy among newcomers to the language, and to these ideas.

So, to summarize:

  • If you want to execute a command numerous times, use a “for” loop.
  • If you have an iterable, and want to create a new iterable, then a list comprehension is probably your best bet.
  • Building a list comprehension is sort of like working in Excel: You start with a set of data, and you create a new set of data. Any expression can be used to map from one to the other.  You don’t care about how Python does things behind the scenes; you just want to get your new data back.
  • A list comprehension consists of either two or three parts, which are often easier to understand if you put them on separate lines: (1) the expression, (2) the data source, and (3) an optional “if” statement.
  • These three lines are analogous to SQL’s SELECT, FROM, and WHERE clauses in a query.  And just as each of those (SELECT, FROM, and WHERE) can use arbitrary expressions, so too can Python’s list comprehensions use arbitrary expressions. A list comprehension will always return a list, though — just as a SELECT will always return a table-like result set.
  • Do you want to create a set, or perhaps a dictionary, rather than a list?  Then you can use a set comprehension or a dict comprehension. The idea is the same as everything I’ve said about list comprehensions, except that your result will be a single set or a single dictionary.

Do you find it difficult to work with list comprehensions?  If so, what’s hard for you about them?  And does the above help to make their use, and their syntax easier to remember?  I’m eager to hear your reactions, so that I can improve these explanations even further.

Why you should almost never use “is” in Python

It’s really tempting, when you first start to use Python, to use “is” rather than “==”.  It’s a bit more readable, and it feels like it should just work, especially when you’re dealing with integers. In a language that uses “or” and “and” instead of “||” and “&&”, it seems logical to use “is” instead of “==”. And if you try “is” with small integers, or even with short strings, you might be lulled into thinking that you should use “is” in lots of places.

But you shouldn’t.  Really, in almost no case, should you use “is”; rather, you should almost certainly use “==”.  In fact, there’s only one case in which most Python programmers should be using “is”, and that’s to check to see if something is None.

In this blog post, which is the result of many questions and discussions I’ve had with students in my Python classes, I’m going to try to describe the reasons for this — and along the way, describe some parts of how Python’s objects are allocated, and what we mean when we say that two objects are “the same.”

Let’s start with the basics: Everything in Python is an object. Every object in Python has a unique ID number, which we can retrieve from an object by using the built-in “id” function:

>>> id(5)
140236457829784

>>> id('abc')
4503718088

>>> id([1,2,3])
4504494160

Now, if two variables are pointing to the same object, they will (not surprisingly) return the same ID:

>>> x = [1,2,3]
>>> y = x
>>> id(x)
4504494160
>>> id(y)
4504494160

Given that x and y point to the same list, changes to the list will be reflected in both variables:

>>> x[0] = '!'
>>> y[1] = '?'
>>> x
['!', '?', 3]
>>> y
['!', '?', 3]

In such a case, it’s pretty clear that x and y are both pointing to precisely the same object. They aren’t just equal in value; they are one and the same — aliases for one another.

We can ask Python if this is true by using the “is” operator, also known as the “identity operator.” “is” doesn’t compare the values of x and y. Rather, it checks to see if x and y have the same ID. If so, then they are the same object. If not, then they aren’t. It’s as simple as that. Perhaps it goes without saying, but two objects that “is” each other are also “==” to each other, since an object’s value should be equal to itself:

>>> x == y
True

>>> x is y
True

>>> id(x) == id(y)
True

The above code shows that x and y have the same ID. This means that they “is” each other; we’re dealing with two names for the same object. Their values are thus equal, which is what “==” checks.

Again: The “is” operator returns “True” if two names are referring to the same object. And the “==” operator returns “True” if two names point to objects that contain the same value.

The most common usage, by far, is when we want to know if something is None. True, we would use “==”. But in both readability and speed, “is None” trumps “== None”. So your code should generally say:

if x is None:
    print("x is None!")

It shouldn’t surprise us to find out that “is” is faster than “==”. After all, “is” is implemented in C, and is a simple comparison of the IDs of the two objects. No function call is needed, and we certainly don’t need to compare the values of the two objects, which can also take some time.

The use of “is None” works because the None object is a singleton in Python. No matter what you do, id(None) will always return the same value. (Note that this value won’t stay constant across different invocations of Python.)  In other words:

>>> id(None)
4315260920

>>> id(None)
4315260920

>>> x = None
>>> id(x)
4315260920

What happens if you try to create a new instance of None? Well, we would first have to find out None’s type:

>>> type(None)
<type 'NoneType'>

Unfortunately, NoneType isn’t a defined identifier in Python:

>>> NoneType
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'NoneType' is not defined

So if we want to create a new instance of None, we’ll need to do it ourselves:

>>> type(None)()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: cannot create 'NoneType' instances

Aha.  Well, that’s a shame. But I was using Python 2.7 in the above example. What if I try Python 3?

>>> type(None)
<class 'NoneType'>

>>> NoneType()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'NoneType' is not defined

>>> x = type(None)()
>>> type(x)
<class 'NoneType'>
>>> x is None
True

So no matter how you slice it, None is a singleton. Which is why you can (and should) use “is None”, rather than “== None”, in your code.

But what happens if you decide that you want to use “is” in other places? The problem is that it will sometimes work. That “sometimes” is because “is” exposes some of Python’s internal optimizations in ways that can be a bit surprising.

Strings are how I was initially introduced to the difference between “==” and “is”, and the danger of using “is” over-zealously. Two equal strings should be “==”, but are they “is”?

>>> x = 'a' * 5
>>> y = 'a' * 5
>>> x == y
True
>>> x is y
True

Well, that’s interesting — and I got the same result in Python 2.7, 3.4, and also in PyPy. But why should this be the case? One possibility is that strings are immutable, and that having Python use a single object for each string that we create, would be efficient. And indeed, this is true — so long as the string is short:

>>>> x = 'a' * 5000
>>>> y = 'a' * 5000
>>>> x == y
True
>>>> x is y
False

The above, which works the same in Python 2.7, 3.4, and in PyPy, demonstrates that Python won’t reuse just any string that we have created. There is a limit.  I experimented with things a bit, and I found that 21 is the magic length at which strings are no longer “is” to one another. That is:

>>> x = 'a' * 20
>>> y = 'a' * 20
>>> x is y
True

>>> x = 'a' * 21
>>> y = 'a' * 21
>>> x is y
False

The above was true in Python 2.7 and 3.4, and also in PyPy. However, I also found some seemingly weird behavior, which is undoubtedly because of the way in which Python byte-compiles and then executes for loops:

>>> for i in range(15,25):
        x = 'a' * i
        y = 'a' * i
        print("[{0}] x is y: {1}".format(i, x is y))

[15] x is y: False
[16] x is y: False
[17] x is y: False
[18] x is y: False
[19] x is y: False
[20] x is y: False
[21] x is y: False
[22] x is y: False
[23] x is y: False
[24] x is y: False

Wow, that’s kind of strange, no? Indeed, in a for loop, I found that the only number for which the two strings were “is” to one another was 1:

>>> for i in range(0,10):
...     x = 'a' * i
...     y = 'a' * i
...     print("[{0}] x is y: {1}".format(i, x is y))
...
[0] x is y: False
[1] x is y: True
[2] x is y: False
[3] x is y: False
[4] x is y: False
[5] x is y: False
[6] x is y: False
[7] x is y: False
[8] x is y: False
[9] x is y: False

At the same time, if you create a long literal string and assign it to a variable, you’ll likely find that the strings are “is” to one another:

>>> x = 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'

>>> y = 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa'

>>> x is y
True

(Forgive the re-formatting that WordPress did to the above assignments; in Python, they were both on one line.)

I’m not sure what is going on here, but it just goes to show that you really shouldn’t use “is” unless you know what you’re doing.  And even if you think that you know what you’re doing, you might still be surprised!  Bottom line: Using “is” on strings is almost always a bad idea.

Now, this is generally something that we don’t need to think or care about very much. But let’s say that you’re working with large strings, and that these strings might repeat themselves on occasion. In such a case, you will end up with many copies of the same string. Python helps us to solve this problem by “interning” strings. Interning is a technique that has been around for many years in the programming world, which allows us to store only one copy of any given string. In Python 2, we use the built-in “intern” function. In Python 3, we must use sys.intern; intern is no longer a builtin.

“intern” takes a string (and only a string) as a parameter. It returns a reference — either to a new string that was created, or to a string that was already allocated. Thus, the length of the string doesn’t matter; even in the case of a long string, it will only be allocated a single time:

>>> from sys import intern     # Python 3 only
>>> x = intern('a' * 5000)
>>> y = intern('a' * 5000)
>>> x is y
True

As you can see, using “intern” guarantees that every unique string is allocated only once. If you use “intern” on the same string a second time, Python returns a reference to the first string.

Python uses “intern” internally for a variety of purposes.  If you’re working with long strings that repeat themselves, then it might be worth using intern. But for the most part, Python creates and allocates so many objects that a few strings here and there are probably not going to make a difference.    Certainly, you should only use “intern” once you have identified bottlenecks.

You might think that even if strings are allocated multiple times, and are thus not “is” to one another, at least integers are going to be identical. After all, Python wouldn’t allocate new objects for numbers, would it?

We can test this pretty easily, of course:

>>> x = 200
>>> y = 200
>>> x is y
True

Well, that’s encouraging, right?  Let’s try something bigger:

>>> x = 2000
>>> y = 2000
>>> x is y
False

So yes, it turns out that even integers that are equal aren’t necessarily pointing to the same object.   As Amy Hanlon pointed out in her fantastic talk about Python “wats”, this is because Python pre-allocates a number of integers. If your integer is within that range, then they will use the same object, and be “is” to one another. But if you’re outside of that range, then you’ll have two separate objects. Unless, of course, you allocate them in the same line of code:

>>> x = 2000; y = 2000
>>> x is y
True

Have I mentioned that you really shouldn’t use “is” to compare objects except for None? I hope that you’re increasingly convinced.

I’ll close this post with a bit of mischief: In theory, if two objects are “is”, then they’re pointing to the same object — which means that they should be identical to one another, and thus also give us a True response to “==”.  While Python doesn’t allow us to redefine “is”, we can redefine what an object says when we try to compare with using “==”:

>>> class Foo(object):
...     def __eq__(self, other):
...         return False
...
>>> f1 = Foo()
>>> f2 = f1
>>> f1 is f2
True
>>> f1 == f2
False

I cannot think of a situation in which this would be a desirable thing to do. But it’s fun, and allows us to sharpen our understanding of the difference between “==” and “is”.

If you liked this explanation, then you’ll likely also enjoy my ebook, “Practice Makes Python,” with 50 exercises meant to improve your Python fluency.