A very sad day — the end of Linux Journal

In 1995, just before moving to Israel, I was working for Time Warner in New York City.  One day, I received email from a company called SSC, publishers of numerous “cheat sheets” for Linux users and programmers. They were about to publish a quick-reference guide for GNU Emacs, and since I was the maintainer of the Emacs FAQ, they wanted my input and thoughts.

I gave them my feedback, and the people at SSC were so thankful for my help that they offered to give me, free of charge, any 10 items from their online catalog. In this way, I got a bunch of quick-reference cards, as well as several issues of the then-new publication, Linux Journal.

I had been using Unix since 1988, and had just started to use this new, open-source version.  (I ordered a distribution Linux from a small company in Connecticut known as “Red Hat,” which made for a fairly straightforward installation on a PC. And by “fairly straightforward,” I mean that it only took a few days of configuration to get X Windows to work on my hardware.)  So I was excited to read through Linux Journal.  I loved what I read, and was in awe of the knowledgeable columnists they had.

In one of those issues was an ad, saying that the magazine would soon be starting a spinoff known as “Websmith,” all about a new phenomenon known as the “World Wide Web.”

Now, I loved to write: I had edited the student newspaper while studying at MIT, and I had always thought that it would be fun to write a column. So I e-mailed the editor of Websmith, asking if they would be interested in having me write a column about CGI programming — which was then the latest and greatest way to create what we now call “dynamic Web content.”  After all, I had been doing Web development since early 1993, back when I set up one of the first 100 sites in the world; I figured that my technical and writing experience could be of help.

The editor said “yes,” and my column — which he called “At the Forge,” fitting in with Websmith’s blacksmith theme — was off to the races.

Websmith didn’t survive for very long; it seems that the World Wide Web wasn’t interesting or big enough to support a business.  (Who knows?  Maybe that’ll change some day.)  But some of us who were writing for Websmith discovered that our columns and articles were simply incorporated into Linux Journal’s next issue, continuing from there.

In other words: I was suddenly and unexpectedly a Linux Journal columnist — a position that I have held, continuously and proudly, since 1996.

It’s thus extremely sad for me to know that Linux Journal is no more.  The magazine’s publisher, Carlie Fairchild, announced the magazine’s closure to the staff and writers earlier this week, and is telling subscribers today.

Just to put things in perspective: Before I met my wife, before my children were born, before I started my 11-year-long PhD program, I was writing for Linux Journal.

For as long as we have been married, my wife has heard the following, typically several times each month, each time with increased urgency: “I have to work on the column, which was due a few days ago. My editor is going to kill me.”

Ditto for my children, whose definition of “deadline” has definitely been affected — for the worse — by my monthly worrying, complaining, and late-night writing.

But I’m not complaining. Writing for Linux Journal for so long has been a defining part of me, and my career, for a very, very long time.

Thanks to LJ, I have had a chance to write, which I so love to do. I have learned a huge amount through my writing, because each column required that I spend time learning before I could teach.

Thanks to LJ, I’ve even gotten some clients, who contacted me as a result of my articles.

Thanks to LJ, my wife and I were invited to visit Alaska and the Caribbean, when I was a featured speaker with “Geek Cruises.”

Thanks to LJ, I got many free books and other resources, was invited to speak at conferences, and was (much to my pleasant surprise) recognized at technical conferences, even when I didn’t speak there.

And of course, thanks to LJ, I’ve gotten to work with some amazing people.  I’ve worked with a variety of editors, the most recent being Jill Franklin, all of whom were talented, helpful, and tolerant of my flexible interpretation of deadlines. They all gave me total freedom to write about whatever topic I wanted, whenever I wanted.  I got to explore all sorts of great topics that were of interest to me, and to my clients, and (I believe) to my readers.

I received e-mail from people all around the world who read my columns, which was deeply satisfying and gratifying.  At conferences, people would see my name badge and tell me that they had been reading my columns for many years.  Even now, as I publish a weekly newsletter for programmers, I often get messages from people saying that they read my column in Linux Journal, and are happy to reconnect with me.

The publishing industry is changing, and that reality is hitting publishers big and small. Indeed, earlier this week, Time Inc. (yes, where I worked when SSC first contacted me) sold itself to another company. The combination of publishing economics, along with the plethora of free, online content for open-source enthusiasts, makes it hard to produce a profitable magazine with top-notch technical content.

Consider the world in 1996: Linux was an oddball operating system, supported by no major manufacturers and seen as a hacker’s plaything. Perl and Python were used by lots of individuals, but not for too many serious applications. MySQL was free, but wasn’t open source, and there weren’t any open-source options for people who wanted to use a database.  And of course, Web applications were in their infancy; my “form-mail” program, which was used by countless sites to send e-mail (before it was taken and modified for the worse by Matt’s Script Archive) remained cutting edge for quite some time.

Things have changed a great deal since then. My phone uses Android, a form of Linux. When the flight attendant reboots the in-flight entertainment system, I see that it’s running Linux, as well. My 14-year-old daughter is learning Python. Companies from Apple to IBM, Cisco to PayPal, Ericsson to VMWare, in the US, Europe, Israel, and China, ask me to teach Python and Git to their employees. Who knew that the technologies I learned, and learned to love, so many years ago, would be so dominant in the world today?

And thus, while Linux Journal might have failed as a business, it succeeded in its mission: To spread the word of Linux and open-source software, to help people to understand, implement, and use open-source technology, and to bring these technologies into the mainstream as a legitimate alternative to the then-dominant commercial offerings.

To my many readers: Thanks for your support over the years. (And you can keep getting weekly writing from me via my “Better developers” newsletter.)

To the amazing staff at Linux Journal, over many years and in many iterations: Thanks for putting up with my delays, and for doing such a great job with my text.  I can’t imagine how they edited a magazine with such in-depth technical content, but they did, and did it well.

To my family: Thanks for putting up with my loud, monthly stresses, including the many times in which I wondered, out loud, how they could possibly think to have a columnist who cannot get his software to work.

Oh, and to the three (!) journalists who e-mailed me in the last 10 days, asking to whom they could pitch stories for Linux Journal: Sorry, folks.  You’re a bit too late.

RIP, Linux Journal.  It has been a fun, wild ride.

Announcing: Trainer Weekly, practical advice for technical trainers

Do you offer technical training?  Or are you just interested in becoming a trainer?

I’ve been doing technical training for several years now, and it has become the main thing that I do.  I love it, and encourage everyone to look into it as a potential career (or part of a career).

I’ve created an online Facebook group for trainers. I’m doing coaching for trainers. And most recently, I started “Trainer Weekly,” a free newsletter for people offering training. Every Monday, you’ll get a new piece of advice about th business, pedagogy, or logistics of training. How do you price things?  How do companies think about training? What sorts of courses can and should you offer?  What kinds of exercises should you give in class?

If this sounds interesting to you, then sign up for Trainer Weekly, and expect to get a new message from me every Monday.  If you have specific questions about training, then just drop me a line; I’ve already addressed a few topics that were requested by readers, and hope to address many more.

Shark Tank increased our traffic by 1000x. Here’s how we handled it.

More than five years ago, I started work to with a small company with a simple idea. Rent Like a Champion was founded by alumni of the University of Notre Dame, which has a very strong culture of college football games. (Note to my non-American readers: We’re talking here about American football, rather than “real” football.)  However, South Bend, Indiana doesn’t have hotels close to the stadium, meaning that people don’t have a nearby, convenient place to stay when they come into town. RLAC thus offers homeowners the chance to rent out their homes on football weekends to visitors. Homeowners make some extra money, and football fans get to see the games they wanted while staying nearby.

I took over for a previous programmer, improving the site’s management and e-commerce features. RLAC has since grown massively, and my role has grown along with them. (And, I hasten to add, I’m no longer doing day-to-day coding on the site; my employee Genadi is doing a fantastic job of that, leaving me to handle more architectural, strategic, and managerial issues on the technology side.) There are still vestiges of the code that I inherited in 2010, but a great deal has changed and improved since then. We are using the latest version of both Ruby and Ruby on Rails, the latest version of MySQL, have both staging and production servers, and have a large number of automated tests. The site rarely goes down or is overwhelmed by traffic.

On the business side, we’re now on several dozen US college campuses, handle incoming and outgoing payments automatically, and even allow people to negotiate on the dates and prices of where they’re staying. Really, I couldn’t be prouder to be associated with this company.

A few months ago, we got some big news: Rent Like a Champion was going to be on Shark Tank, a reality TV show that allows you to pitch your company to investors. We didn’t know what this would mean for the company’s future, but we did know that it would mean lots of additional traffic. The numbers we heard were in the area of 10-20 thousand requests per second. This, from a site that had only 10-20 requests per second on our busiest days. Meaning, we had to figure out a reasonable way to scale our site up so that it could handle about 1,000 times the traffic we were used to seeing.

Now, Rails has a reputation for not being scalable. But it’s clear that Rails can scale, given enough computers. When someone says that it’s “not scalable,” what they probably mean is that given a certain amount of traffic, Rails requires more servers than other languages and frameworks. And yes, that’s probably true. But we clearly weren’t going to change our entire technology stack just for a few hours of television time.

We thus took a multi-pronged approach, working on every part of the site — from the infrastructure, to the application, to the servers we were using.

The bottom line, by the way, was that we more than survived the ordeal: Not only did we handle about 8,000 simultaneous requests/second for an extended period of time, but our servers were barely breaking a sweat. And as if that weren’t enough, we got investments from both Mark Cuban and Chris Sacca, two well-known billionaire investors.

So, what did we do to scale up? And what does this mean for projects you’re doing?

No state on the server

First and foremost, modern Web applications can be inherently scalable, if you design them correctly. Rails, like many other modern frameworks, assumes a “zero state” situation on the Web application server. This means that no user-related state is stored on the server itself. Instead, we store all such state in the database. This means that we can add as many servers as we want, because the actual data won’t be stored on the Web server.

One of the potentially tricky parts has to do with user sessions. Every modern Web framework offers developers the chance to use sessions; the user’s cookie contains an ID (typically encrypted) that allows us to look on disk, in memory, or in a database for the user’s session information.

If the session information is stored in a disk file, then you can only use a single Web server. That’s because if the same user visits several servers in the same day (which happens all of the time, if you have several servers behind a load balancer), and the session info is spread across multiple servers, they’ll effectively be logged out when they reach the second server. You can avoid this by storing session information in the database, but then that’ll slow down the application.

We decided to use cookie-based session information, in which the cookie itself contains the user’s session info. That means we don’t have to worry about how many servers we’re using, because the session info will be available to all of them. If you only store a user ID in the session, then the cookie will be particularly tiny (albeit encrypted), and thus you don’t have to worry about the size.

One part of our scaling was thus to ensure that users could move freely from one server to another. We then put all of the Web servers behind a single load balancer, giving the illusion that RentLikeAChampion.com was a single machine, but actually distributing the load among many. The user’s cookie would go from their browser, to the load balancer, to the ultimate machine. From there, an ActiveRecord lookup via the user ID gave the user’s information.

Bottom line: Your app might already be more scalable than you think, if session information is stored in cookies. If you’re using files to store session information, though, your application is inherently non-scalable; once you use more than one Web server, the user might end up being logged out.

Use VMs, but not necessarily AWS

Several of the biggest trends we’ve seen in the last few years involve the combination of cloud computing and virtual machines. Thanks to VMs, we can spin up as many servers as we need, only when we need them.

I’ve long been of the opinion that putting a server “in the cloud,” as everyone likes to say nowadays, is often unnecessary. There’s nothing wrong with a plain ol’ server, after all, especially since such servers are often going to be cheaper and easier to maintain. (I’m getting to the point where I might be changing my tune, as I see the advantages of deploying to a new VM or container with each release, rather than upgrading each existing release.)

However, in the case of RLAC, moving to the cloud was a no-brainer — not because we need it for our permanent infrastructure, but because the key worry we had was being able to scale up quickly. We didn’t know how many requests we would get, and putting in place a set of “real” servers that could handle that capacity would cost a fortune. Besides, we knew that we needed to scale up quickly for (and during) the Shark Tank airing, but then scale down just as quickly following the show’s broadcast.

Amazon Web Services is the first name that people think of in this space. We decided to go a different route, in no small part thanks to the suggestions and connections of RLAC’s CIO, Mike Hostetler: We went with Server Central, a Chicago-based company that has massive bandwidth, and lots of experience configuring virtual machines for this kind of work. Sure, AWS might be better known. But one of the advantages we got from Server Central was actual, in-person help — something that Amazon wouldn’t provide for a player as small as ourselves.

I have to say that Server Central’s staff amazed and impressed me at every turn: They were available, helpful, and polite, and knew a ton about how to configure and tune our VMs for maximum benefit. They set up our MariaDB cluster (more on that below), as well as a load balancer. They helped us to clone VMs ahead of schedule, and to connect them (virtually, of course) with our network.

In the end, we didn’t go with a Chef or Puppet configuration of our VM, but rather used a combination of Capistrano to deploy our software our main VMs, which we then cloned for some backup VMs. Not super elegant, I’ll admit, but it worked just fine, and meant that we could eliminate another learning curve. Mike H. worked extensively with Server Central, and really got the hang of configuring and deploying those VMs, such that we had more than 20 available for the night of broadcast.

Bottom line: If you want to scale up and/or down quickly, then VMs are almost certainly the way to go. And if you’re looking to get actual service, rather than a faceless SaaS company, I’d definitely suggest speaking with Server Central.

Switching from Apache to nginx

I’ve said it on many occasions: I have a warm spot in my heart for htApache httpd. I’ve been doing Web development since before the first version of Apache was released, much less the Apache Foundation was founded, and I have always found it to be an easy to use, flexible HTTP server.

However, there’s no doubt that another server, nginx, offers greater performance and scalability than Apache. In the case of RLAC and Shark Tank, we were far more interested in scalability than ease of use. However, we didn’t want to have a super-hard learning curve and transition, which we feared would be the case if we moved to a combination of nginx and Unicorn, a popular duo in the Ruby on Rails world. We thus settled on using nginx and Phusion Passenger, a plugin that provides Ruby on Rails applications which we had previously used with Apache.

I have to admit that I was pleasantly surprised on all fronts by nginx. It installed without a hitch, thanks in no small part to Passenger’s super-easy installation process on Linux boxes. The configuration is quite different from Apache, but not as difficult as I would have expected. It allowed us to use our existing SSL certificates quite easily. And the performance, without a doubt, was quite impressive. Indeed, any performance issues we saw were the result of Passenger, rather than nginx; Ruby and Rails both use lots of memory, and thus there is a limit to the number of simultaneous requests you can handle per machine.

Bottom line: I hate to say it, but I don’t see a big advantage to Apache any more. nginx documentation and tutorials are quite good, support for Rails applications is excellent with Passenger, and the performance we saw quite very good. Configuration of Apache is still easier for me, but that’s going to be true for anything I’ve been doing for 20 years, I expect.

Database cluster

With the Web servers scaling up nicely, the biggest potential bottleneck suddenly became our database. The database on RLAC, as with most Web applications, is involved in every single page displayed, from showing the user’s name to listing homes for a particular football game, to letting homeowners set the prices for individual games and events.

The problem is that the database is a finite resource; if too many people come and visit the site at the same time, the database will cease responding to some of them, causing a domino effect that will cause many people to get errors or timeouts.

I’m a big fan of PostgreSQL, which I have been using for about 20 years. But when I inherited the RLAC code, we were already using MySQL on the site — and switching technologies is almost never a good idea if things are already working, so I didn’t move things around.

While PostgreSQL’s master-master replication is still being discussed and designed, MySQL has master-master, high-availability clustering working already. Even better, following the purchase of MySQL by Oracle (via their purchase by Sun), Monty Widenius, the original author of MySQL, has been working on MariaDB, a MySQL-compatible plugin that offers superior performance.

Server Central suggested that we use a HA master-master cluster of VMs, all running MariaDB. From our perspective as developers, it looked and felt like a normal MySQL database. But the performance was rock solid and super fast, and it meant that we didn’t need to worry about the database being a bottleneck, unless we were crushed by people actually trying to use the application.

Importing our old database (from the MySQL server on Rackspace) to our new MariaDB cluster was fast and easy. The two database servers feel almost identical, and the dump-restore cycle took a matter of minutes.

Bottom line: I feel somewhat chastened; after years of telling people how much better it is to use PostgreSQL, and how master-slave fallover is probably fine for most purposes, I was pleasantly surprised to use a master-master cluster that more than suited our needs. PostgreSQL could probably have handled the load just as well, of course, but switching to a different database just before a major PR event would have been a very bad idea.

Caching

The above was a great way to get our system to work under high loads. But it’s always possible to optimize things further, and one of the best-known ways to make Web applications increase their speed is to add caching. If you can cache a page, then everyone benefits: The user gets a must faster response, and the server doesn’t have to spend its time running Ruby and SQL, because the request has been served without even touching the server.

On RLAC, we used three different types of caches. These worked together spectacularly well, improving our performance massively. Our working assumption was that we would have a number of VMs on standby in case we would need them, and that most users would look at a few pages before bouncing off of our site. But we still knew that we would need to server thousands (and many tens of thousands) of requests per second, many of which just wanted to see our home page while watching Shark Tank on their TVs.

Our caches were as follows:
  • In our application, we cached many of our SQL queries. (Ruby on Rails lets you do this inside of your application with very little fuss.) Thus, every time we asked the database for the list of homes at event #12345, we stored that information in a cache, in case that same event would be requested again in the near future.
  • On each of our servers, we (mostly Mike Hostetler) installed and configured Varnish, which provided us with caching of static assets, such as images, CSS, and JavaScript. Moreover, the configuration meant that if someone came to the RLAC home page without being logged in, they would get a cached version of the page. If, however, they came after being logged in, they would hit our Rails app. This was necessary to ensure that people who had logged. We had a separate copy of Varnish running on each of our VMs, which was admittedly not the best solution, but it worked more than well enough for our purposes. It meant that every HTTP request coming to a VM from the load balancer would first go through Varnish, and only if necessary go to the Rails server.  I should add that Varnish is one of the most impressive pieces of software I’ve seen in a long time; it’s very good at what it does, and has an incredibly powerful configuration language that lets you examine and rewrite HTTP queries in amazing ways.  It’s definitely worth looking into, if you haven’t already.
  • Finally, we used CloudFlare, a content distribution network (CDN) that did initial caching of static assets, including all of our images, JavaScript, and CSS. One of the nicest parts of CloudFlare is that they have an “emergency page” feature, so that if your site isn’t available, you can at least show a decent static page, and ask people to return later.

Bottom line: We could have optimized our caching even more than this. But after a lot of configuration of Varnish, CloudFlare, nginx, and the Rails app’s HTTP response headers, we got to the point where each of our VMs could handle about 2,000 simultaneous requests that hit the database, and far more than that if they were static and only required the home page. But the caching was incredibly helpful.

Images on S3

This isn’t directly related to the move to Server Central and getting ready for Shark Tank, but it certainly helped: Many of our images, and especially photos of homes available for rent, are now stored on Amazon’s S3, rather than our own server. This reduces the load on our VM and nginx, and provides users with much faster retrieval speeds than we can provide. Moving things to S3 turns out to be quite easy to do; most of the image-uploading gems now available for Rails have an option to store things in S3; once we set up the appropriate buckets, this was a piece of cake.

Bottom line: The cost of S3 storage is laughably low, because you’re just paying for the bandwidth, and the per-byte cost is basically a joke. It’s totally reasonable to do this, and you’ll end up spending far less than you ever thought.

Our biggest mistake: delayed jobs

Remember how I said every VM was stateless, and that it didn’t matter how many VMs we would run? Yeah, that’s what we thought — and then we started to migrate our systems, and remembered that our Web server was configured to run Delayed Jobs. DJ is a Ruby gem that works with Rails, which does just as the name describes — it allows you to offload things to the background, so that your users (and server) don’t sit around. For example, RLAC processes many photos from homeowners. That processing, and subsequent copying of images to S3, can take some time. So we have offloaded it to DJ, which runs in the background all of the time, processing images.

About three days before we were supposed to switch over to Server Central, with our many VMs, we realized that Delayed Jobs expected files to be uploaded to a specific server. That is, we had state — files on the system that DJ was going to look for, scoop up, and process. DJ uses the database, which meant that it seemed we had to choose between rewriting our app such that all image uploads would go to a single, designated server, or that we would upload to many servers, with DJ (on all but the server that received the file) and the database going crazy because it cannot find the uploaded file on the local filesystem.

After some discussion (and a bit of panic), my developer Genadi found the answer: DJ lets you designate a queue name when the system starts up. We thus set up our DJ startup script, such that it grabbed the hostname on which it was running, and used that hostname as the name of its DJ queue. Thus, with 20 servers, we had 20 separate DJ queues, each of which was from a particular machine. This wasn’t the most efficient solution; if we had thought about it earlier, we probably would have centralized things, with a single DJ server. But it worked beautifully, required very little configuration (except for figuring out how to start up DJ with this option via our startup scripts), and pulled us through.

Bottom line: Think very hard about whether you have any extra services running that might involve state.

Cron jobs

Finally, the RLAC system has a large number of processes that are run via cron jobs. These are crucial for the business, in that they send out reminder e-mail messages to homeowners and renters, handle credit cards, and make homeowner payments. You don’t want these to go down, but you also don’t want these to run on every machine. Clearly, in contrast with DJ, we didn’t want these cron jobs to run on each of our Web servers.

The solution was to designate one of our servers, and to set it up to use cron. Thus, while our servers are all on identical VMs, you could say that one is more equal than others, in that it is running a variety of rake tasks via cron to deal with system maintenance. Perhaps we’ll pull these tasks onto a separate VM one of these days, but for now, things are working quite well.

The only (minor) hitch was that RLAC’s e-mail servers are run by Google, and Gmail started to tag all of the e-mail produced by our cron jobs as spam, and rejecting it. The solution was to configure Postfix (our SMTP server) to use SendGrid, our outgoing e-mail provider. The moment we did that (which was shockingly easy to do, I must admit), I started to get reports from our cron jobs once again.

Bottom line: If you have regularly scheduled cron jobs, either for system features or for maintenance (e.g., checking disk space or making database backups), make sure that you know in advance where you’ll be putting them. And test things, to make sure that you’re able to receive e-mail from your cron jobs, in case of success or failure.

Team effort

Perhaps the most important aspect of what we did was that it was a team effort.  None of us could have done this by ourselves; even on a relatively small application, there were too many types of responsibility and expertise for anyone to have done it on their own. I really enjoyed getting to work with not only Genadi Reznichenko (the amazing developer who has worked for me for more than three years), but also Mike Hostetler (who took on the role of CIO over the last year), and Trevan Hetzel (our front-end developer), as well as the RLAC team led by CEO Mike Doyle.  Communication and trust are the most important aspects of any software project, and we did well on both fronts, thanks to GitHub, Slack, weekly group phone calls, and lots of pair-programming sessions.

We worked hard to get things up and running; it was a team effort, but one which paid off in spades for the company. I hope that this description helps some of you to think about scaling up your Web applications in various ways; if you have stories to share, or questions (or comments) about what I’ve written here, please let me know. I’d love to swap stories and/or hear what you went through!

Updated ebook: Jewish Guide to Visiting China

cover-loresI’m happy to announce that my ebook, “The Jewish Guide to Visiting China,” has been updated quite a bit — and now has its own Web site, JewishChinaGuide.com.

If you’re Jewish and planning to visit China in the near future, I’m sure that my book will help to remove some of your concerns and fears, while giving you lots of practical advice that I’ve gleaned over my 10 trips to Beijing, Shanghai, and Nanjing.

In addition to updating the book itself a great deal, I have also added a new deluxe package, which includes not only the book, but a 30-minute Skype/phone call with me, in which I’ll address the questions and concerns you have about keeping kosher and Shabbat while in China.

In addition to this ebook, I continue to publish Mandarin Weekly, a free weekly newsletter containing the best links I can find for people (like me!) studying Mandarin Chinese.  You can get Mandarin Weekly sent to you via e-mail at the Web site, or follow it on Twitter, at @MandarinWeekly.

Registration is open for my October Webinars (about regexps and technical training)

September has been busy with work and holidays, but I’m gearing up for an exciting and busy October. Among other things, I’m giving two (free) Webinars in that month, and you can already register for them:

  • Intermediate Regular expressions: In my previous Webinar about regular expressions, I covered the basics.  In this one, we’ll go further, spending a great deal of time talking about groups, backreferences, and some other topics that tend to confuse people.  I’ll be using Python to demonstrate regexps, but this Webinar isn’t only aimed at Python developers. Registration is free; sign up here.
  • Technical training: I’m starting to spend more and more time helping people to become technical trainers, teaching programming in high-tech companies. (Indeed, I’m in the process of starting my coaching program for people interested in improving their training skills.) In this Webinar, which I expect will be the first of many, I’ll give an overview of the technical-training landscape, how it works, and why you should seriously consider providing training services. I’ll briefly review pedagogical, logistical, and business considerations for the aspiring technical trainer. Registration for this training is free; you can sign up here.

Now available: My interview on “Developer on Fire” about training programmers

I was recently interviewed by Dave Rael for the “Developer on Fire” podcast. That episode has now been released, at http://developeronfire.com/Podcast/Episodes/reuven-lerner-sharing-insight. Enjoy!

Announcing: Coaching for technical trainers

What, exactly, do I do for a living?

Yes, I’m a programmer — or as I’m supposed to say nowadays, a “full-stack Web developer.” And yes, I’m a lead developer/CTO.  And yes, I’m a writer, with two ebooks written (about Python and visiting China), two more on the way, and my monthly Linux Journal column now in its 20th (!) year.

But over the last few years, another role has increasingly dominated the others: Much of my time is now spent as a technical trainer, teaching a variety of open-source technologies — Python, Ruby, Git, and PostgreSQL — to companies around the world.

Some software developers believe that training is less demanding, less fulfilling, or even less lucrative than creating software. And for some of them, that might well be true.

I have personally found training to be no less demanding than developing software — but I have also found that it is extremely fulfilling, and that it does a more-than-adequate job of paying the bills. We all know that high-tech companies are desperately looking for high-quality developers; when I do my job right, I provide them with such developers, ready to use the latest technologies to solve problems more efficiently and reliably than would otherwise have been possible.

The good news is that my training business is going quite well; I’m now booked solid for several months in advance, and I get to work with some great companies and very bright engineers. Part of my motivation for writing ebooks is now to reach the people whom I cannot possibly teach in person.

Being a trainer requires a number of skills beyond knowing how to program: You also need to know how to organize a syllabus, prepare exercises, prepare slides and printed materials, speak in front of a group, and answer questions. Beyond that, you also need to understand the business side of training — what are companies looking for, how do you approach them, how much do you charge them, and how can you grow your training business when companies are satisfied with your work. These skills, like many others, take time develop, and everyone has their own strengths and weaknesses. But I believe that if you’re willing to put in the effort, then you can learn how to become a technical trainer, and have the same sense of job satisfaction that I do.

If you are such a person, interested in offering technical training — on any subject, not just the technologies in which I specialize — then I invite you to consider joining my coaching program for technical trainers. This isn’t a course; it’s a personalized program that will help you to improve, month by month, in all of the ways that you need to become successful. You’ll have access not just to me, but to other trainers with varying levels of experience, who will provide you with feedback — just as I hope you’ll help them. The program is still in its infancy, but I believe that I’ve put together a combination of resources that can help everyone to become a trainer.

I’m launching the coaching program in two weeks, on October 1st, 2015. There aren’t any formal start of finish times; if you want to start later, then that’s fine, as well. My hope is that you’ll stay in the program for as long as you need to improve, getting feedback from me and others.  I also hope and expect that the program will more than pay for itself.

I’ll be holding a free Webinar on the subject of technical training on October 14th, at which I’ll also be taking questions from anyone who might be new to this field, or be curious about what it involves. I invite you to read more about my coaching program, to contact me if you have any questions about it, and to register for the Webinar that I’ll be holding next month!

New ebook: Jewish guide to visiting China

Jewish Guide to Visiting ChinaAs many people know, I’ve visited China seven times over the last three years, traveling there to give courses in Python and Ruby. I just got back from my most recent trip, and found it to be as fun and exciting as ever. You could say that I’ve gotten a bit obsessed with the country; I read books about China, have been taking daily Chinese lessons since August, and publish a free weekly newsletter (Mandarin Weekly) with links to useful resources for people learning Chinese.

Given that I keep kosher and Shabbat, other religious Jews are increasingly asking me for advice on what, where, and how they can be Jewishly observant when visiting China on business or pleasure. No one in China is likely to know or care about such subjects, let alone know anything about Judaism, so it can be a bit daunting to visit there for the first time.

I’ve collected my advice into a 40-page ebook, the “Jewish guide to visiting China.” If you’re a religiously observant Jew who will be visiting China for short periods of time, then I believe this guide can significantly reduce the time (and stress) you’ll need to invest before your trip.

I’m just launching it now — and for the first week it’s online, I’m offering a discount coupon (“YouTaiRen” — aka 犹太人 — the word for “Jew” in Chinese) giving 20% off of the normal $6 price. This price includes PDF, Mobi, and ePub formats, which should suit any computer or ebook reader.

Again: The Jewish guide to visiting China, now available for 20% off with the “YouTaiRen” offer code.

I expect that the book will expand significantly over time; if you purchase this book, you’re automatically entitled to updates and upgrades.

I welcome comments, suggestions, and additions!

How not to write an error message

Users and programmers see error messages very differently.

When a user sees an error message, they think, “Oh, no.  Something went wrong.”  Rarely, in my experience, does the user think to read the error message, or to use it as a clue toward what might have happened.  Technology is so opaque, so hard to understand, and so seemingly random that people have stopped paying attention to error messages.  The situation hasn’t been helped by the “blue screen of death,” or messages that say, “Error -16,” or the like.  The more people get such messages, the more they’re likely to tune them out.

When clients ask me for help with a problem, I often ask them to tell me what error message they saw. This question often surprises them; they don’t expect that the error message will tell them what their next step could be. But in good software, it will.

Programmers thus see error messages as helpful hints toward how a solution can be solved. (Of course, programmers and engineers are generally interested in solving problems — something that the public at large doesn’t necessarily share.)  When I see an error message, I ask, “Hmm, let’s see if this is telling me what I can do to remove the problem.” I see the error message as a helpful hint that the system’s programmer left for me, rather than an insult or yet more proof that technology is fickle.

But it turns out that when programmers are new or inexperienced with a technology, they often exhibit the same behavior as novices. This week, I taught a two-day course in Git. Now, Git is not exactly known for being an easy-to-understand system — but as I showed my students, there are many times when the error message does tell us something useful. Perhaps it’s using sui generis Git-flavored technobabble, but once you understand how Git works behind the scenes, the message will actually make sense.

My point is that even if error messages are ignored by the majority of users, they can and should be written such that people can benefit from them. Even if the number of people who will read these messages is small, you should at least give them a hint, and indicate why things when wrong.

Indeed, a good error message should say: (1) what inputs were provided that caused this error, (2) what was wrong with those inputs, and (3) what you can do to solve the problem. It’s totally fine to log lots of additional information to a logfile or database for further analysis later on. But to the degree that it’s possible, error messages should try to help people to help themselves.

That’s nice in theory, of course, In reality, there are the completely, maddeningly, stupid error messages that give all programmers a bad name.

Case in point: About two years ago, I bought a 12-trip train ticket (“kartisiya”) from Modi’in (where I live) to Bnai Brak (where I had a client). I figured that I would make enough trips to Bnai Brak that it was worth getting these tickets, and putting them on my electronic ticket card, known in Israel as a “Rav kav.” But it turned out that it was actually faster for me to get off at one of the Tel Aviv train stations, and then walk to this client’s office.

A few days ago, I decided that if I hadn’t used these tickets in two years, I should probably get them refunded. Or changed to another city. Or something.  Unfortunately, because they were on my Rav Kav, no station would refund my money. Nor could they change the tickets to be between Modi’in and another city. I was stuck with 12 tickets to a train station that I wasn’t likely to need or use in the near future.

Fortunately, Israel Railways has a fancy Web site. I went there, and found that I could submit questions to their help line via a Web form. Note that Israel Railways has removed phone numbers and e-mail addresses from their “contact us” page; if you want to contact them, you’re encouraged to (1) use a Web form, (2) send a fax, or (3) send postal mail.

So I went to the Web form, filled in my story about my 12-trip ticket to Bnai Brak, provided lots of personal details, and was greeted with the following message:

train-fail

If you don’t read Hebrew, then here’s my quick transaction of the central text:

Form for public queries

Dear customer,

The system wasn’t able to handle your query.

Thanks for contacting us!

So, which is it? Are they thanking me?  Or telling me that something went wrong?  (Or both?) And did something go wrong because of a failure on my part, or on theirs?  Should I re-submit my query?  (I did; I got the same failure a second time.)

In the end, I discovered that there is a live-chat option on the Israel Railways site. Surprisingly enough, it was staffed by a real human, who asked me what was wrong.  I told her what I wrote above — and she said that complaint has been registered, and someone will get back to me in the near future.

I believe that they will get back to me. And I believe (hope) that I’ll be able to get my money back on those tickets.

But I keep thinking about the programmer who implemented this error message, and how he or she has, without knowing it, contributed to the general public’s sense that technology is random, unfriendly, and destined to drive us mad.

Are you a programmer? If so, do you consider the content of your error messages when you write them, and what people will do with them if and when they appear? Remember that error messages are meant to be read by people, not machines — and try to write them accordingly.

“Practice Makes Python” is now available for early-bird purchase

My first ebook, “Practice Makes Python” — containing 50 exercises that will help to sharpen your Python skills — is now available for early-bird purchase!3D_book

The book is already about 130 pages (and 26,000 words) long, containing about 40 exercises on such subjects as basic data structures, working with files, functional programming, and object-oriented development. But it’s not quite done, and thus I’m calling this an “early-bird” purchase of the book: Not all of the exercises are ready, the formatting isn’t quite there yet, and PDF is the only format available for now. That said, even in this draft version, there is more than enough here to help many Python developers to gain fluency and improve their skills with the language.

Anyone who purchases the book now can use the coupon code EARLY to get a 10% discount. Perhaps it goes without saying, but anyone buying the book now will also get all updates and improvements, free of charge, as they occur over the coming weeks. And anyone who finds that they didn’t get value from the book is welcome to e-mail me and say so — and I’ll refund 100 percent of your purchase price.

The basic idea behind “Practice Makes Python” is that learning Python — or any language — is a long, slow process. Even the best courses cannot possibly give you enough practice with the language for it to feel natural. That only comes with practice. Most people end up practicing, as it were, on projects at work. My goal with this book is to give people who have taken Python courses a chance to become more familiar with the language.

My PhD studies in Learning Sciences taught me a great deal about how people learn, and one of the most important lessons was that of “constructionism” — that one of the best ways to learn is through the creation of things that are important to the individual. I have tried to make the exercises in “Practice Makes Python” interesting and fun, as well as relevant to what people do with Python on a day-to-day basis. Perhaps you won’t be creating Pig Latin translation programs in your day job, but the techniques that you learn from writing such programs in the book will undoubtedly help you out. Certainly, by working through the exercises — not by reading the answers and discussions! — you will learn a great deal about Python programming.

If you recently took a course in Python, or even if you have been working with it for up to a year, I believe that “Practice Makes Python” will give you the knowledge and confidence you need to master this fun and interesting language. These exercises are based on the many Python courses I have taught in the United States, Europe, Israel, and China over the years, and have proven themselves to help programmers start to really “get” Python.

I’d be delighted to hear what you think about “Practice Makes Python,” and how it can help to improve people’s Python programming skills even more. Contact me at reuven@lerner.co.il if you have thoughts or ideas.