Answering Python questions from readers

Every so often, I’ve asked readers of my free, weekly “Better developers” newsletter to send me their Python problems. And every so often, I get a chance to answer their questions, going through their Python problems and trying to solve them.

I’ve recently recorded and uploaded two videos with solutions to their problems, which I’m sharing here.

Have Python problems you want to solve? Send them to me at reuven@lerner.co.il — and if I choose your question, I’ll give you 30% off any course in my online catalog.

Question 1: Finding links in an e-mail newsletter, and saving them to an Excel spreadsheet

Question 2: Simple PostgreSQL queries from Python


Python’s str.isdigit vs. str.isnumeric

Let’s say that I want to write some Python code that invites the user to enter a number, and then prints that number, tripled. We could say:

>>> n = input("Enter a number: ")
>>> print(f"{n} * 3 = {n*3}")

The good news is that this code works just fine. The bad news is that it probably doesn’t do what you might expect. If I run this program, I’ll see:

Enter a number: 5
5 * 3 = 555

The reason for this output is that the “input” function always returns a string. So sure, we asked the user for a number, but we got the string ‘5’, rather than the integer 5. The ‘555’ output is thanks to the fact that you can multiply strings in Python by integers, getting a longer string back. So ‘a’ * 5 will give us ‘aaaaa’.

Of course, we can always create an integer from a string by applying the “int” class to the user’s input:

>>> n = input("Enter a number: ")
>>> n = int(n)
>>> print(f"{n} * 3 = {n*3}")

Sure enough, we get the following output:

Enter a number: 5
5 * 3 = 15

Great, right? But what happens if the user gives us something that’s no longer numeric? The program will blow up:

Enter a number: abcd

ValueError: invalid literal for int() with base 10: 'abcd'

Clearly, we want to avoid this problem. You could make a good argument that in this case, it’s probably best to run the conversion inside of a “try” block, and trap any exception that we might get.

But there’s another way to test this, one which I use with my into Python classes, before we’ve covered exceptions: Strings have a great method called “isdigit” that we can run, to tell us whether a string contains only digits (0-9), or if it contains something else. For example:

>>> '1234'.isdigit()
True

>>> '1234 '.isdigit() # space at the end
False

>>> '1234a'.isdigit() # letter at the end
False

>>> 'a1234'.isdigit() # letter at the start
False

>>> '12.34'.isdigit() # decimal point
False

>>> ''.isdigit() # empty string
False

If you know regular expressions, then you can see that str.isdigit returns True for ‘^\d+$’. Which can be very useful, as we can see here:

>>> n = input("Enter a number: ")
>>> if n.isdigit():
n = int(n)
print(f"{n} * 3 = {n*3}")

But wait: Python also includes another method, str.isnumeric. And it’s not at all obvious, at least at first, what the difference is between them, because they would seem to give the same results:

>>> n = input("Enter a number: ")
>>> if n.numeric():
n = int(n)
print(f"{n} * 3 = {n*3}")

So, what’s the difference? It’s actually pretty straightforward, but took some time for me to find out: Bascially, str.isdigit only returns True for what I said before, strings containing solely the digits 0-9.

By contrast, str.isnumeric returns True if it contains any numeric characters. When I first read this, I figured that it would mean decimal points and minus signs — but no! It’s just the digits 0-9, plus any character from another language that’s used in place of digits.

For example, we’re used to writing numbers with Arabic numerals. But there are other languages that traditionally use other characters. For example, in Chinese, we count 1, 2, 3, 4, 5 as 一,二,三,四, 五. It turns out that the Chinese characters for numbers will return False for str.isdigit, but True for str.isnumeric, behaving differently from their 0-9 counterparts:

>>> '12345'.isdigit()
True

>>> '12345'.isnumeric()
True

>>> '一二三四五'.isdigit()
False

>>> '一二三四五'.isnumeric()
True

So, which should you use? For most people, “isdigit” is probably a better choice, simply because it’s more clearly what you likely want. Of course, if you want to accept other types of numerals and numeric characters, then “isnumeric” is better. But if you’re interested in turning strings into integers, then you’re probably safer using “isdigit”, just in case someone tries to enter something else:

>>> int('二')
ValueError: invalid literal for int() with base 10: '二'

Just when I thought I was done with this, David Beazley reminded me that there’s a third method I should be dealing with: str.isdecimal. This asks a slightly different question, namely whether we have something which is a number but not a decimal number. What does this mean?

Well, in Python, we can describe “2 to the 2nd power” as “2 ** 2”. But if you want to print it, you’ll probably use something a bit fancier, such as 2². This is a two-character string, in which the first character is ‘2’ and the second character is ‘²’. The second one contains digits, but it’s not described as a decimal number. Thanks to Unicode, you can create such strings either via the keyboard or using Unicode-entry syntax. Thus:

>>> s = '2²'    # or if you prefer, s = '2' + '\u00B2'

>>> s.isdigit()
True

>>> s.isnumeric()
True

>>> s.isdecimal()
False

Most of us, most of the time, can thus use these three methods interchangeably, with little chance of being mixed up. Once you start using all sorts of interesting Unicode-based numbers, things can get a bit weird and interesting.

I tried to determine whether there was any difference in speed between these methods, just in case, but after numerous tests with “%timeit” in Jupyter, I found that I was getting roughly the same speeds from all methods.

If you’re like me, and ever wondered how a language that claims to have “one obvious way to do it” can have multiple seemingly identical methods… well, now you know!

Beyond the “hello, world” of Python’s “print” function

One of the first things that anyone learns in Python is (of course) how to print the string, “Hello, world.”  As you would expect, the code is straightforward and simple:

print('Hello, world')

And indeed, Python’s “print” function is so easy and straightforward to use that we barely give it any thought.  We assume that people know how to use it — and for the most part, for most of the things they want to do, that’s true.

But lurking beneath the surface of the “print” function is a lot of functionality, as well as some history (and even a bit of pain).  Understanding how to use “print” can cut down on the code you write, and generally make it easier for you to work with.

The basics

The basics are simple: “print” is a function, which means that if you want to invoke it, you need to use parentheses:

>>> print('hello')

hello

You can pass any type of data to “print”. Strings are most common, but you can also ints, floats, lists, tuples, dicts, sets, or any other object. For example:

>>> print(5)

5

or

>>> print([10, 20, 30])

[10, 20, 30]

And of course, it doesn’t matter whether the thing you’re trying to print is passed as a literal object, or referenced by a variable:

>>> d = {'a':1, 'b':2, 'c':3}>>> print(d)

{'a':1, 'b':2, 'c':3}

You can also put an expression inside of the parentheses; the value of the expression will be passed to “print”:

>>> print(3+5)
8

>>> print([10, 20] + [30, 40])
[10, 20, 30, 40]

Every object in Python knows how to display itself as a string, which means that you can pass it directly to “print”. There isn’t any need to turn things into strings before handing them to “print”:

print(str([10, 20, 30])    # unnecessary use of "str"

[10, 20, 30]

After “print” displays its output, it adds a newline.  For example:

>>> print('abc')
>>> print('def')
>>> print('ghi')
abc
def
ghi

You can pass as many arguments as you want to “print”, separated by commas. Each will be printed, in order, with a space between them:

>>> print('abcd', 'efgh', [10, 20, 30], 99, 'ijkl')

abcd efgh [10, 20, 30] 99 ijkl

We’ll see, below, how we can change these two default behaviors.

Inputs and outputs

If “print” is a function, then it must have a return value. Let’s take a look:

>>> x = print('abcd')
>>> type(x)
NoneType

In other words: “print” returns None, no matter what you print. After all, you’re not printing in order to get a return value, but rather for the side effect.

What about arguments to “print”?  Well, we’ve already seen that we can pass any number of arguments, each of which will be printed.  But there are some optional parameters that we can pass, as well.

The two most relevant ones allow us to customize the behavior we saw before, changing what string appears between printed items and what is placed at the end of the output.

The “sep” parameter, for example, defaults to ‘ ‘ (a space character), and is placed between printed items.  We can set this to any string, including a multi-character string:

>>> print('a', 'b', 'c', sep='*')
a*b*c

>>> print('abc', 'def', 'ghi', sep='***')
abc***def***ghi

>>> print([10, 20, 30], [40, 50, 60], [70, 80, 90], sep='***')
[10, 20, 30]***[40, 50, 60]***[70, 80, 90]

Notice that “sep” is placed between the arguments to “print”, not between the elements of each argument.  Thus in this third example, the ‘***’ goes between the lists, rather than between the integer elements of the lists.

If you want the arguments to be printed alongside one another, you can set “sep” to be an empty string:

>>> print('abc', 'def', 'ghi', sep='')
abcdefghi

Similarly, the “end” parameter defaults to ‘\n’ (newline), but can contain any string. It determines what’s printed after “print” is done.

For example, if you want to have some extra lines after you print something, just change “end” so that it has a few newlines:

>>> def foo():
        print('abc', end='\n\n\n')
        print('def', end='\n\n\n')
>>> foo()
abc


def


If, by contrast, you don’t want “print” to add a newline at the end of what you print, you can set “end” to be an empty string:

>>> def foo():
        print('abc', end='')
        print('def', end='')

>>> foo()
abcdef>>>

Notice how in the Python interactive shell, using the empty string to print something means that the next ‘>>>’ prompt comes after what you printed.  After all, you didn’t ask for there to be a newline after what you wrote, and Python complied with your request.

Of course, you can pass values for “end” that don’t involve newlines at all. For example, let’s say that you want to output multiple fields to the screen, with each field printed in a separate line:

>>> def foo():
        print('abc', end=':')
        print('def', end=':')
        print('ghi')

>>> foo()
abc:def:ghi

Printing to files

By default, “print” sends its data to standard output, known in Python as “sys.stdout”.  While the “sys” module is automatically loaded along with Python, its name isn’t available unless you explicitly “import sys”.

The “print” function lets you specify, with the “file” parameter, another file-like object (i.e., one that adheres to the appropriate protocol) to which you want to write. The object must be writable, but other than that, you can use any object.

For example:

>>> f = open('myfile.txt', 'w')

>>> print('hello')
hello
>>> print('hello???', file=f)
>>> print('hello again')
hello again
>>> f.close()

>>> print(open('myfile.txt').read())
hello???

In this case, the output was written to a file.  But we could also have written to a StringIO object, for example, which acts like a file but isn’t one.

Note that if I hadn’t closed “f” in the above example, the output wouldn’t have arrived in the file. That’s because Python buffers all output by default; whenever you write to a file, the data is only actually written when the buffer fills up (and is flushed), when you invoke the “flush” method explicitly, or when you close the file, and thus flush implicitly. Using the “with” construct with a file object closes it, and thus flushes the buffers as well.

There is another way to flush the output buffer, however: We can pass a True value to the “flush” parameter in “print”.  In such a case, the output is immediately flushed to disk, and thus written.  This might sound great, but remember that the point of buffering is to lessen the load on the disk and on the computer’s I/O system. So flush when you need, but don’t do it all of the time — unless you’re paid by the hour, and it’s in your interest to have things work more slowly.

Here’s an example of printing with and without flush:

>>> f = open('myfile.txt', 'w')
>>> print('abc', file=f)
>>> print('def', file=f)
>>> print(open('myfile.txt').read())  # no flush, and thus empty file

>>> print('ghi', file=f, flush=True)  
>>> print(open('myfile.txt').read())  # all data has been flushed to disk
abc
def
ghi

You might have noticed a small inconsistency here: “print” writes to files, by default “sys.stdout”. And if we don’t flush or close the file, the output is buffered.  So, why don’t we have to flush (or close, not that this is a good idea) when we print to the screen?

The answer is that “sys.stdout” is treated specially by Python. As the Python docs say, it is “line buffered,” meaning that every time we send a newline character (‘\n’), the output is flushed.  So long as you are printing things to “sys.stdout” that end with a newline — and why wouldn’t you be doing that? — you won’t notice the buffering.

Remember Python 2?

As I write this, in January 2019, there are fewer than 12 months remaining before Python 2 is no longer supported or maintained. This doesn’t change the fact that many of my clients are still using Python 2 (because rewriting their large code base isn’t worthwhile or feasible).  If you’re still using Python 2, you should really be trying to move to Python 3.

And indeed, one of the things that strikes people moving from Python 2 to 3 would be the differences in “print”.

First and foremost, “print” in Python 2 is a statement, not an expression. This means that the parentheses in 2 are optional, while they’re mandatory in 3 — one of the first things that people learn when they move from 2 to 3.

This also means that “print” in Python 2 cannot be passed to other functions. In Python 3, you can.

Python 2’s “print” statement didn’t have the parameters (or defaults) that we have at our disposal.  You wanted to print to a file other than “sys.stdout”?  Assign it to “sys.stdout” to use “print” — or just write to the file with the “write” method for files.  You wanted “print” not to descend a line after printing?  Put a comma at the end of the line.  (Yes, really; this is ugly, but it works.)

What if you’re working in Python 2, and want to get a taste of Python 3’s print function?  You can add this line to your code:

from __future__ import print_function

Once you have done so, Python 3’s “print” function will be in place.

Now I know that Python 3 is no longer in the future; indeed, you could say that Python 2 is in the past. But for many people who want to transition or learn how to do it, this is a good method. But watch out: If you have calls to “print” without parentheses, or are commas to avoid descending a line, then you’ll need to do more than just this import.  You will need to go through your code, and make sure that it works in this way. So while that might seem like a wise way to to, it’s only the first step of a much larger transition from 2 to 3 that you’ll need to make.

Enjoyed this article?  Join more than 11,000 other developers who receive my free, weekly “Better developers” newsletter. Every Monday, you’ll get an article like this one about software development and Python:



 

Registration ends today for Weekly Python Exercise

Weekly Python Exercise logoIf you want to join this month’s cohort of Weekly Python Exercise, you’d better act fast: Registration ends today!

  • If you feel stuck after having taken a Python course or read a book, and don’t know how to improve,
  • If you want to get real-world practice problems that’ll help you with your career,
  • If you want to stop searching Stack Overflow each time you have to use Python,
  • If you want to benefit from a community of learners who assist one another

Then Weekly Python Exercise is a great way to go.

Click here to join Weekly Python Exercise.

Questions?  You can always e-mail me (reuven@lerner.co.il) or hit me up on Twitter (@reuvenmlerner).  Or you can watch the Q&A Webinar I held last night, answering questions from others interested in joining this cohort:

And don’t forget that I offer discounts for students, pensioners/retirees, and people living outside of the world’s 30 wealthiest countries.

Hundreds of other developers have improved their Python skills with WPE in the last 18 months.  Improve your career, stop feeling stuck, and stop searching Stack Overflow, and improve your Python fluency!

Reminder: Weekly Python Exercise registration ends soon

Weekly Python Exercise logoIf you want to improve your understanding of Python, then you’re going to have to practice.

And as hundreds of developers from around the world have already learned, there’s no better way to practice than Weekly Python Exercise.

If you’re relatively new to Python, this our upcoming cohort is for you! Weekly Python Exercise: Newbie Edition, will be starting soon.  Enjoy 15 weeks of beginner-level exercises (and solutions), along with a private forum for collaboration, and regular office hours with me.

Click to learn more, including seeing a free sample of WPE and how it works.

I’m excited about this new cohort — and if you want to get better at Python, then I hope you’ll join me!

 

Announcing a new course: Intro Python — Fundamentals

Python is one of the hottest languages out there. People can’t get enough Python, and companies can’t get enough Python people.

This means that learning Python is a great move for your career.  (Also, it’s just plain ol’ fun to use.)

If you’ve always wanted to get started with Python, or if you’ve been using it by combining good guesses with many visits to Stack Overflow, then I’m happy to announce the release of my new course, “Intro Python: Fundamentals.”

This course is meant for experienced programmers with up to six months of Python experience.  It covers the language’s basic syntax, and the core data structures: Numbers, strings, lists, tuples, dicts, and sets.   And of course, it uses the current (3.7) version of Python.

The course has nearly 7 hours of video (in 79 lectures), plus 11 exercises to practice and improve your Python fluency.   It’s the same material I cover on the first day of the four-day intro Python class that I give to companies around the world.

You can read more about the course here.   As with all of my courses, I offer discounts to students, retirees/pensioners, and people living outside of the world’s 30 richest countries — just contact me at reuven@lerner.co.il for the appropriate discount code.

So if you’ve ever wanted to learn Python, or just want to strengthen your understanding of how Python’s core data structures work, take a look at “Intro Python: Fundamentals“.

Announcing: Two Python courses in Tel Aviv

Good news for developers in Israel — I’ll be offering two open-enrollment courses in Tel Aviv next month (January):

  • Advanced Python: If you’ve already been using Python for at least six months, and want to level up your knowledge of this hot language, then this course is for you.  I’ll talk about advanced data structures, inner functions, functional programming techniques, in-depth objects, iterators, generators, decorators, and threading/multiprocessing.
  • Intro to data science and machine learning: Data science is the hottest topic in the computer world, and Python is the most popular language for doing it!  This course will teach you about NumPy, Pandas, and scikit-learn, using a variety of real-world data sets.  Learn how to import, select, clean, and analyze data — and then to create and compare machine-learning models that make predictions.  Note that this course assumes some familiarity with Python.

These are the same courses I’ve given on-site, around the world, to companies such as Apple, Cisco, Ericsson, IBM, Intel, and VMWare (among many others).  The big difference is that now you don’t have to work at a big company to take the course.

Want more information, or to guarantee a spot?  Just click on the above links.

Have any questions? Send me e-mail at reuven@lerner.co.il, or call me at +972-54-496-8405.

Announcing: Weekly Python Exercise, Newbie Edition

  • Weekly Python Exercise logoAre you new to Python, and looking for relatively easy exercises to reinforce what you’ve learned?
  • Do you use Python every day, but rely on Stack Overflow to answer questions like, “Which parentheses should I use here,” or, “What’s the difference between a list and dict?”
  • Are you an experienced developer in other languages, and want to learn Python via hands-on practice?

If any of the above is true, then I’m excited to announce enrollment for my latest course: Weekly Python Exercise, Newbie Edition.

Since I launched it 18 months ago, three cohorts of students have participated in Weekly Python Exercise — receiving a new Python challenge via e-mail every Tuesday, and the solution the following Monday.  Students had access to our exclusive forums, and traded ideas, solutions, and techniques with one another. Some attended my live, video office hours, when I answered Python questions that they might have.

The question I got most often about those Weekly Python Exercise cohorts was whether it was appropriate for someone totally new to Python.  Many queries later, I decided to launch Weekly Python Exercise, Newbie Edition.

If you want to improve your Python fluency, then there’s no better way than practice. And if you are new to Python, and want to understand the core concepts better, then I’m sure that Weekly Python Exercise is for you.

WPE Newbie Edition costs $75. What do you get for that?

  • 15 exercises, sent to you each Tuesday — along with a list of subjects covered by the exercise, and links to resources to read on those subjects
  • When applicable, automated tests (in “pytest”) to help you check your answers
  • 15 detailed explanations, sent to you the following Monday
  • Exclusive access to a forum in which other WPE students will help one another to solve the problems (and improve on my solutions)
  • Exclusive access to monthly office hours with me, in which I’ll solve the exercises and answer your Python problems.

And of course, my usual discounts for students, retirees/pensioners, and people from outside the world’s 30 richest countries get significant discounts. (Just contact me to get the appropriate coupon code.)

Sound interesting? Learn more here.

Have any questions? Send me e-mail (reuven@lerner.co.il), or hit me up on Twitter (@reuvenmlerner).

Last chance to save 40% on Python and Git courses

Today (Monday) is the last day to benefit from my weekend sale, with 40% off of my books and courses!  Just enter the coupon code “BF2018” for any product , and take 40% off.

This offer includes my new “Intro Python: Fundamentals” course, containing the 1st day of the 4-day training I give to experienced developers around the world.

Here are the courses I’m currently offering:

Any questions?  Just e-mail me at reuven@lerner.co.il.  But don’t delay; today’s the last day to take advantage of this sale!

Black Friday sale — improve your Python + Git for 40% off!

Yup — just like everyone else, I’m having a Black Friday sale on all of my online books and courses. Just this weekend, you can use the coupon code BF2018 to improve your Python skills:

Just use the coupon code BF2018 at checkout to get your 40% discount!  But hurry, the sale only lasts through “Cyber” Monday.  Any questions?  Just e-mail me at reuven@lerner.co.il.