Subscribe: Ned Batchelder's blog
Added By: Feedage Forager Feedage Grade A rated
Language: English
answer  candy  code  didn  element  interesting  iterable  list  names iterable  names  python  result  things  time  values  work 
Rate this Feed
Rate this feedRate this feedRate this feedRate this feedRate this feed
Rate this feed 1 starRate this feed 2 starRate this feed 3 starRate this feed 4 starRate this feed 5 star

Comments (0)

Feed Details and Statistics Feed Statistics
Preview: Ned Batchelder's blog

Ned Batchelder's blog

Ned Batchelder's personal blog.


What’s in which Python 3.4–3.6?


This is the third in a series of summarizations of what is in each release of Python. The first two were What’s in which Python 2.x? and What’s in which Python 3.0–3.3?.

3.4: March 16, 2014

  • pip is always available, via ensurepip
  • asyncio (provisional API)
  • enum
  • Other stdlib modules: statistics, pathlib, and tracemalloc

Full list of 3.4 changes.

3.5: September 13, 2015

  • async and await syntax
  • matrix multiplication operator @
  • more unpacking generalizations
  • The typing module for type hints
  • os.scandir()

Full list of 3.5 changes.

3.6: December 23, 2016

  • f-strings
  • kwargs and class attributes order is preserved
  • dicts happen to be (but are not guaranteed to be) ordered
  • underscores in numeric literals
  • variable annotations
  • secrets module in stdlib

Full list of 3.6 changes.

A Python gargoyle


In the #python IRC channel today, someone asked:Does anyone know of any libraries that could convert ‘1-5,7,9,10-13’ to [1,2,3,4,5,7,9,10,11,12,13] ?This seemed like an interesting challenge, and people started offering code. This was mine. It’s ungainly and surprising, and I wouldn’t want to keep it, so I call it a gargoyle:[     i for p in s.split(',')     for a, _, b in [p.partition('-')]     for i in range(int(a), int(b or a)+1) ] There are a few things going on here. First, this is a list comprehension, but with three nested loops. A simple list comprehension has this form:result = [ EXPR for NAMES in ITERABLE ] which is the same as this code:result = [] for NAMES in ITERABLE:     result.append(EXPR) For many, the list comprehension seems kind of backwards, where the expression comes first, before the loop produces it. Then the multi-loop form can seem like another surprise:result = [ EXPR for NAMES1 in ITERABLE1 for NAMES2 in ITERABLE2 ] which is equivalent to:result = [] for NAMES1 in ITERABLE1:     for NAMES2 in ITERABLE2:         result.append(EXPR) The first time I ever tried to write a double list comprehension, I thought the loops should go in the other order, in keeping with the Yoda-style of EXPR coming first. They don’t.Back to the gargoyle. It’s a triply-nested loop (this time formatted a little differently):[     i      for p in s.split(',')         for a, _, b in [p.partition('-')]             for i in range(int(a), int(b or a)+1) ] The first loop splits the number ranges on comma, to produce the individual chunks: ‘1-5’, ‘7’, ‘9’, ‘10-13’. This seems like an obvious first step.The next loop is the most surprising. Strings in Python have a .partition method, which is super-handy and under-used. It takes a separator, and produces three values: the part of the string before the separator, the separator itself, and the part of the string after the separator. The best thing about .partition is that is always produces three values, even if the separator doesn’t appear in the string. In that case, the first value is the whole string, and the other two values are empty strings:>>> '1-5'.partition('-') ('1', '-', '5') >>> '12'.partition('-') ('12', '', '') >>> This means we can always assign the result to three names. Super-handy.But list comprehensions can’t have assignments in them, so what to do? Recently, the Python-Ideas mailing list had a thread about adding assignments to list comprehensions, which can simplify complicated comprehensions. In that thread, Stephan Houben pointed out that you can already get the same effect with a cute trick:for x in [17]:     do_something() # has the same effect as: x = 17 do_something() We can explicitly make a one-element list, and “iterate” over it to assign its value to a name. In my gargoyle, we get a and b as the two numbers in the chunk.The third loop is where we actually generate numbers. We’ll use range(), and we always want to start from the first number in the chunk. If there was a second number in the chunk (b), then we want to iterat[...]

Coverage 4.5


Just out: v4.5.

There’s one new feature: configurator plug-ins, that let you run Python code at startup to set the configuration for coverage. This side-steps a requested feature to have different exclusion pragmas for different versions of Python.

People wanted to be able to say, this line of code is excluded from coverage when run under Python 3, but not under Python 2. That sounds simple enough, but then some wanted to be able to exclude for Python 3.5, but not 3.6. Or, excluded when running under PyPy, but not under CPython.

I could see this turning into a never-ending road of finer and finer differentiation. Next would be operating systems, or versions of Django, or, etc, etc, etc.

Rather than me building all that into coverage itself, now you can write your own plug-in that makes all those determinations, and sets the exclude pragmas as you like.

Python’s misleading readability


One of the things that has made Python successful is its readability. Code is clear and easy to understand. One of the reasons is that Python uses words for a few things that other languages use symbols for. But sometimes the readability is misleading. Beginners construct valid Python expressions that don’t do what they seem like they should do.Let’s say you want to know if your variable x is equal to 17. You could do:if x is 17: This might work. But then if you try:if name is "Ned": it doesn’t work? What!? Why not? It’s so clear.The problem is that “is” doesn’t check two values for equality, it checks if the left and right side are precisely the same object. But you can have two different string objects, each of which has the value “Ned”. You don’t use “is” to check equality, you use “==”:if name == "Ned": It’s not just strings: numbers can also do surprising things:>>> 1000 + 1 is 1001 False “x is 17” is more English-like than “x == 17”, but it isn’t right. This is one time that Python’s famed readability leads you to the wrong construct.Another example: you need to know if the answer was either “y” or “yes”, so you try this:if answer == "y" or "yes":     print("Thanks") and now your program doesn’t work. No matter what answer is, it prints “Thanks.” Why?The “or” operator is for combining boolean (true/false) expressions. The result is true if either of its operands is true. So your code is equivalent to:if (answer == "y") or ("yes"):     print("Thanks") If answer is “y”, then the if will be true. If answer isn’t “y”, then the or will consider the right-hand side, “yes”. Strings are true if they are not empty, so “yes” is always true. So the if condition will always be true, no matter what value answer has.The right ways to do this are:if answer == "y" or answer == "yes":     print("Thanks") or if you want to be fancier,if answer in {"y", "yes"}:     print("Thanks") (a list or a tuple would also work here instead of a set, though then you get into philosophical debates about how many data structures fit on the head of a pin.)Don’t get me wrong, I agree that Python is very readable. And every language has constructs that seem like they should work, but don’t. You have to study well, and be careful to use your chosen language properly. [...]



When I was in high school, I was not a diligent student of English. I would try to do the reading on the subway ride, and not very well. One morning, I was on page 38 when I got to my stop. I didn’t have a bookmark, and I didn’t like to turn down the corners of pages, so to remember my place, I thought about what was interesting about the number 38.

Hmm, 38 is twice a prime. It’s one more than 37, which is a prime. Oh, and 39 is three times a prime. Interesting: a sequence of three consecutive numbers that are respectively 1-times, 2-times, and 3-times a prime. BTW, it worked: to this day, I remember what page I was on!

Then a recent tweet brought back this interesting confluence: 2018 has the same property. 2017 is a prime, 2018 is twice a prime, and 2019 is three times a prime.

It turns out this property is rare. There are only eleven numbers less than 3000 like this: 14, 38, 158, 542, 878, 1202, 1382, 1622, 2018, 2558, 2858; and only 541 such numbers less than 1,000,000.

So 2018 is a truly unusual year. 2017 was difficult politically, but there are hopeful signs of a turning tide. Here’s to the year ahead, and fighting the good fights: personally, locally, and nationally.

Iter-tools for puzzles: oddity


It’s December, which means Advent of Code is running again. It provides a new two-part puzzle every day until Christmas. They are a lot of fun, and usually are algorithmic in nature.One of the things I like about the puzzles is they often lend themselves to writing unusual but general-purpose helpers. As I have said before, abstraction of iteration is a powerful and under-used feature of Python, so I enjoy exploring it when the opportunity arises.For yesterday’s puzzle I needed to find the one unusual value in an otherwise uniform list. This is the kind of thing that might be in itertools if itertools had about ten times more functions than it does now. Here was my definition of the needed function:def oddity(iterable, key=None):     """     Find the element that is different.     The iterable has at most one element different than the others. If a     `key` function is provided, it is a function used to extract a comparison     key from each element, otherwise the elements themselves are compared.     Two values are returned: the common comparison key, and the different     element.     If all of the elements are equal, then the returned different element is     None.  If there is more than one different element, an error is raised.     """ The challenge I set for myself was to implement this function in as general and useful a way as possible. The iterable might not be a list, it could be a generator, or some other iterable. There are edge cases to consider, like if there are more than two different values.If you want to take a look, My code is on GitHub (with tests, natch.) Fair warning: that repo has my solutions to all of the Advent of Code problems so far this year.One problem with my implementation: it stores all the values from the iterable. For the actual Advent of Code puzzle, that was fine, it only had to deal with less than 10 values. But how would you change the code so that it didn’t store them all?My code also assumes that the comparison values are hashable. What if you didn’t want to require that?Suppose the iterable could be infinite? This changes the definition somewhat. You can’t detect the case of all the values being the same, since there’s no such thing as “all” the values. And you can’t detect having more than two distinct values, since you’d have to read values forever on the possibility that it might happen. How would you change the code to handle infinite iterables?These are the kind of considerations you have to take into account to write truly general-purpose itertools functions. It’s an interesting programming exercise to work through each version would differ.BTW: it might be that there is a way to implement my oddity function with a clever combination of things already in itertools. If so, let me know! [...]

Bite-sized command line tools: pylintdb


One of the things I love about Python is the abundance of handy libraries to cobble together small but useful tools. At work we had a large pylint report, and I wanted to understand it better. In particular, I wanted to trace back to which commit had introduced the violations. I wrote to do the work.

Since we had a lot of violations (>5000!) I figured it would take some time to use git blame to find the commit for each line. I wanted a way to persist the progress through the lines. SQLite seemed like a good choice. It also would give me ad-hoc queryability, though to be honest, I didn’t even consider that at the time.

SQLite is part of the Python standard library, but there’s a third-party library that makes it super-convenient to use. Dataset lets you use a database without creating a schema or even model first. You just open a database, choose a table name, and then start writing dictionaries to it. It handles all the schema creation (or modification!) behind the scenes. Awesome.

These days, click is the tool of choice for command-line parsing, and other chores needed in the terminal. I used the progress bar functions. They aren’t perfect, but in only a few lines I had a workable indicator.

Other useful things from the Python standard library:

  • concurrent.futures for parallelizing the git blame work. It’s got a high-level “map” interface that did exactly what I needed without having to think about queues, threads, and so on.
  • subprocess.check_output does the subprocess thing people usually want: just run the command and give me the output.

pylintdb isn’t earth-shattering, it just does exactly what I needed in 120 lines with a minimum of fuss, thanks to dataset, click, and Python.

New design


I’ve just updated the design of this site. The goal was to make it responsive, so that it would work well on small screens, but I made other changes along the way. The body type is now serif rather than sans, and much larger. I made lots of other tweaks as I worked on pages.

Making a responsive design was fun: it meant working out mechanisms for the layout rather than just a static design.

Of course, it’s easy to get carried away. Take a look at what happens to my name in the header when the screen gets below 300 pixels: Ned Batchelder becomes nedbat to save space. This was accomplished with the help of a span with class “chelder”...

It took me a long time to make this design. I started it 15 months ago, but stopped work on it for more than a year. I picked it up again two weeks ago, and powered through the remaining work.

Behind the scenes, I changed only one thing: using Sass to generate the CSS. The rest is still as janky and difficult as always.

For comparison (and posterity), here is the design I just replaced. If anything seems amiss with the new design, just let me know.

Candy in my pocket


Let me tell you a modern-day horror story: for almost ten days, I didn’t have a phone!

I was on a day trip to Montréal, and my phone just completely died. I thought maybe it just needed a charge, but nope, nothing would bring it back. I had a nicely sculpted chunk of glass.

(Side note: I had been texting with Susan, so eventually I dashed off a quick email: “My phone is completely dead. I can’t even tell what time it is.” She sent back an email that said just, “It’s 11:24.” Is it any wonder I love her?)

At first, I felt a bit lost. I couldn’t take pictures, I couldn’t use maps, I couldn’t text with Susan to coordinate getting picked up at the airport.

But what I noticed is that much of what I was used to doing with my phone, I didn’t really miss. I didn’t have games to jump to when I had a free moment. I wasn’t able to reflexively look up interesting tidbits. I couldn’t anxiously check if I had gotten an interesting email.

I realized I didn’t need those things. It was OK to not have a smaller screen to use when I wasn’t using my larger screen. I started to feel like the phone was like a bag of candy in my pocket. If my pocket is full of candy, I’ll nibble on candy all day long. But when I didn’t have a bag of candy, I didn’t miss it. Sure, sometimes I’d like to have a piece of candy, but not nearly as much as I ate when I always had a bag of candy with me.

Now I finally (USPS can be disappointing...) have a new phone, a Google Pixel. I’ll be glad to have my podcasts back. I can take pictures again. I was lucky not to need a two-factor app in the last week.

I’ll have to see how I relate to it. I’ll have the candy with me, but will I be able to resist nibbling? I wasn’t as bad as some: I never had the impulse to read my phone while standing at a urinal, for example. But I nibbled a lot more than it turns out I needed to. I’ll try to keep in mind how these last ten days felt, and snack responsibly.

Finding your first OSS project


Running in the circles I do, I often hear the question, “Where’s a good open source project to start off contributing to?” The last time this came up, I asked on Twitter and got some good replies.

The best answers pointed to two aggregators of projects. These sites collect links to projects that have special labels for bug reports that are good for first-time contributors to work on. The presence of these labels is a good indicator that the project is well-maintained, welcoming to newcomers, and prepared for their contributions.

  • Up For Grabs lists dozens of projects, helpfully showing how many open first-timer issues each has.
  • Awesome for Beginners is lower-tech, but also lists projects with links to their first-timer tagged issues.

I also got links to some useful advice for first-time contributors:

Making a first contribution can be overwhelming. Keep looking through these resources until you find something that makes it feel do-able.