Subscribe: Ned Batchelder's blog
Added By: Feedage Forager Feedage Grade A rated
Language: English
code  coverage  data  def  find  import  module  ned library  part  people  python  reverse  run  seed  steps  test  tests 
Rate this Feed
Rate this feedRate this feedRate this feedRate this feedRate this feed
Rate this feed 1 starRate this feed 2 starRate this feed 3 starRate this feed 4 starRate this feed 5 star

Comments (0)

Feed Details and Statistics Feed Statistics
Preview: Ned Batchelder's blog

Ned Batchelder's blog

Ned Batchelder's personal blog. 4.3.2 and 4.3.3, and 4.3.4


A handful of fixes for today: v4.3.2. Having active contributors certainly makes it easier to move code more quickly.

...and then it turns out, 4.3.2 wouldn't run on Python 2.6. So quick like a bunny, here comes version 4.3.3.

...and then that fix broke other situations on all sorts of Python versions, so version 4.3.4.

Evil ninja module initialization


A question about import styles on the Python-Dev mailing list asked about imports like this:import os as _os Understanding why people do this is an interesting lesson in how modules work. A module is nothing more than a collection of names. When you define a name in a .py file, it becomes an attribute of the module, and is then importable from the module.An underlying simplicity in Python is that many statements are really just assignment statements in disguise. All of these define the name X:X = 17 def X(): print("look!") import X When you create a module, you can make the name "X" importable from that module by assigning to it, or defining it as a function. You can also make it importable by importing it yourself.Suppose your module looks like this:# import os def doit():     os.something_or_other() This module has two names defined in it: "doit", and "os". Someone else can now do this:# from yourmodule import os # or worse, this imports os and doit: from yourmodule import * This bothers some people. "os" is not part of the actual interface of yourmodule. That first import I showed prevents this leaking of your imports into your interface. Importing star doesn't pull in names starting with underscores. (Another solution is to define __all__ in your module.)Most people though, don't worry about this kind of name leaking. Import-star is discouraged anyway, and people know not to import os from other modules. The solution of renaming os to _os just makes your code ugly for little benefit.The part of the discussion thread that really caught my eye was Daniel Holth's winking suggestion of the "evil ninja mode pattern" of module initialization:def ninja():     global exported     import os     def exported():         os.do_something() ninja() del ninja What's going on here!? Remember that def is an assignment statement like any other. When used inside a function, it defines a local name, as assignment always does. But an assignment in a function can define a global name if the name is declared as global. It's a little unusual to see a global statement without an explicit assignment at the top-level, but it works just fine. The def statement defines a global "exported" function, because the global statement told it to. "os" is now a local in our function, because again, the import statement is just another form of assignment.So we define ninja(), and then execute it immediately. This defines the global "exported", and doesn't define a global "os". The only problem is the name "ninja" has been defined, which we can clean up with a del statement.Please don't ever write code this way. It's a kind of over-defensiveness that isn't needed in typical Python code. But understanding what it does, and why it does it, is a good way to flex your understanding of Python workings.For more about how names (and values) work in Python, people seem to like my PyCon talk, Python Names and Values. [...]

No PyCon for me this year


2017 will be different for me in one specific way: I won't be attending PyCon. I've been to ten in a row:


This year, Open edX con is in Madrid two days later after PyCon, actually overlapping with the sprints. I'm not a good enough traveler to do both. Crossing nine timezones is not something to be taken lightly.

I'll miss the usual love-fest at PyCon, but after ten in a row, it should be OK to miss one. I can say that now, but probably in May I will feel like I am missing the party. Maybe I really will watch talks on video for a change.

I usually would be working on a presentation to give. I like making presentations, but it is a lot of work. This spring I'll have that time back.

In any case, this will be a new way to experience the Python community. See you all in 2018 in Cleveland!

D'oh: 4.3.1


Yesterday I released five months' of fixes as 4.3, and today I am releasing 4.3.1. This is not because releasing is fun, but because releasing is error-prone.

Two bad problems were very quickly reported by my legions of adoring fans, and they are now fixed. I'll sheepishly tell you that one of them was a UnicodeError in a bit of too-cute code in

Perhaps I should have released a 4.3 beta. But my experience in the past is that betas do not get the kind of attention that final releases do. Partly this is just due to people's attention budget: lots of people won't install a beta. But it's also due to continuous integration servers. When a final release is out, hundreds if not thousands of CI servers will install it automatically as part of the next triggered build. They won't install pre-releases.

So there's a not-great choice to make: should I put out a beta, and hope that people try it and tell me what went wrong? Will enough people in enough disparate environments take that step to truly test the release?

Or should I skip that step, jump straight to a final release, and prepare instead to quickly fix whatever problems occur? I chose the latter course for 4.3. I guess I could use meta-feedback about which form of feedback I should pursue in the future... 4.3


The latest release: 4.3 is ready.

This version adds --skip-covered support to the HTML report, implements sys.excepthook support, reads configuration from tox.ini, and contains improvements that close 18 issues. The complete change history is in the source.

A special shout-out to Loïc Dachary: he read my blog post about Who Tests What, and got interested in contributing. And I mean, really interested. Suddenly he seemed to be everywhere, making pull requests and commenting on issues. In a week, I had 122 emails due to his activity. That energy really helped push me along, and is a big reason why this release happened, five months after 4.2.

Random trivia: this is the 30th version on PyPI; it's the 57th if you include pre-releases.

Finding test coupling


Before we get started: this is a story about a problem I had and how I solved it. This retelling is leaving out lots of small false trails and hard learnings, which I will summarize at the end. I report these stories not to lecture from on high, but to share with peers, help people learn, and ideally, elicit teachings from others so that I can do it better next time. The main qualities I am demonstrating here are not intelligence and experience, but perseverance, patience, and optimism.OK, on with the story:Running our large test suite the other day, we got a test failure. It seemed unrelated to the changes we were making, but you can never be sure, so we investigated. Along the way I used a few techniques to narrow down, widen, and identify suspects.Running just that one test passed, but running the whole test suite, it failed, and this behavior was repeatable. So we had some kind of coupling between tests. Ideally, all tests would be isolated from each other. Perfect test isolation would mean that no matter what order you ran tests, and no matter what subset of tests you ran, the results would be the same. Clearly we did not have perfect test isolation.The job now was to find the test we were coupled with, or perhaps one of the many possible tests that we were coupled with.The test failure itself was a UnicodeError while trying to log a warning message involving a username with a non-ASCII character in it. Apparently this is something that doesn't work well: when warnings are routed through the logging system, if the message is actually logged, and the message has a non-ASCII Unicode string, an exception will happen. That's unfortunate, but we'll have to live with that for the moment.Our best guess at the moment is that when the test passes, it's because either the warnings settings, or the logging settings, are deciding not to log the warning. When the test fails, it's because some previous test has changed one (or both!) of those settings, causing the message to proceed all the way through the warnings/logging pipeline, to the point of producing the UnicodeError. This is a plausible theory because those settings are global to the process, and would be easy to change without realizing the consequences for test suites.But we still have to find that test. Here's the command that runs just the one test, that failed:python ./ lms test --verbosity=2 --with-id --settings=test \     --xunitmp-file=/edx/app/edxapp/edx-platform/reports/lms/nosetests.xml \     --with-database-isolation \     openedx/core/djangoapps/external_auth/tests/ This is the Django test runner, using nose. That last line selects one particular test method in one particular class in one specific test file. To try to find a failing combination, we'll widen the scope of our test run by peeling off trailing components. This will give us progressively more tests in the run, and eventually (we hope), the test will fail:openedx/core/djangoapps/external_auth/tests/ openedx/core/djangoapps/external_auth/tests/ openedx/core/djangoapps/external_auth openedx/core/djangoapps This last one finally failed, with 1810 tests. That's still too many to examine manually. We can run those tests again, with nose-randomly to randomize the order of the tests. This gives us an opportunity to run experiments where the randomization can tell us something about coupling. If we run the 1810 tests, and our failing test doesn't fail, then none of the tests that ran befor[...]

Dragon iterators


Advent of Code is running again this year, and I love it. It reveals a new two-part Santa-themed puzzle each day for the first 25 days in December. The puzzles are algorithmically slanted, and the second part is only revealed after you've solved the first part. The second part often requires you to refactor your code, or deal with growing computational costs. I've long been fascinated with Python's iteration tools, so Day 16: Dragon Checksum was especially fun.Here's an adapted version of the first part of the directions: You'll need to use a modified dragon curve. Start with an appropriate initial string of 0's and 1's. Then, for as long as you need, repeat the following steps: Call the data you have at this point, A. Make a copy of A; call this copy B. Reverse the order of the characters in B. In B, replace all instances of 0 with 1 and all 1's with 0. The resulting data is A, then a single 0, then B. For example, after a single step of this process, 1 becomes 100. 0 becomes 001. 11111 becomes 11111000000. 111100001010 becomes 1111000010100101011110000. We have a few options for how to produce these strings. My first version took an initial seed, and a number of steps to iterate:ZERO_ONE = str.maketrans("01", "10") def reverse01(s):     """Reverse a string, and swap 0 and 1."""     return s.translate(ZERO_ONE)[::-1] def dragon_iterative(seed, steps):     d = seed     for _ in range(steps):         d = d + "0" + reverse01(d)     return d (BTW, I also wrote tests as I went, but I'll omit those for brevity. The truly curious I'm sure can find the full code on GitHub.) This is a simple iterative function.The problem statement sounds like it would lend itself well to recursion, so let's try that too:def dragon_recursive(seed, steps):     if steps == 0:         return seed     else:         d = dragon_recursive(seed, steps-1)         return d + "0" + reverse01(d) Both of these functions have the same downside: they produce complete strings. One thing I know about Advent of Code is that they love to give you problems that can be brute-forced, but then turn up the dials high enough that you need a cleverer algorithm.I don't know if this will be needed, but let's try writing a recursive generator that doesn't create the entire string before returning. This was tricky to write. In addition to the seed and the steps, we'll track whether we are going forward (for the first half of a step), or backward for the second half:def dragon_gen(seed, steps, reverse=False):     if reverse:         if steps == 0:             yield from reverse01(seed)         else:             yield from dragon_gen(seed, steps-1, reverse=not reverse)             yield "1"      &#x[...]

Who Tests What


The next big feature for is what I informally call "Who Tests What." People want a way to know more than just what lines were covered by the tests, but also, which tests covered which lines.This idea/request is not new: it was first suggested over four years ago as issue 170, and two other issues (#185 and #311) have been closed as duplicates. It's a big job, but people keep asking for it, so maybe it's time.There are a number of challenges. I'll explain them here, and lay out some options and questions. If you have opinions, answers, or energy to help, get in touch.First, it's important to understand that works in two main phases, with an optional phase in the middle: The first phase is measurement, where your test suite runs. notes which code was executed, and collects that information in memory. At the end of the run, that data is written to a file. If you are combining data from a number of test runs, perhaps for multiple versions of Python, then there's an optional combination phase. Multiple coverage data files are combined into one data file. The reporting phase is where your project is analyzed to understand what code could have run, and the data files are read to understand what code was run. The difference between the two is the code that did not run. That information is reported in some useful way: HTML, XML, or textually. OK, let's talk about what has to be done...MeasurementThe measurement phase has to collect and record the data about what ran.What is Who?At the heart of "Who Tests What" is the Who. Usually people want to know what tests run each line of code, so during measurement we need to figure out what test is being run.I can see two ways to identify the test being run: either figures it out by examining function names being run for "test_*" patterns, or the test runner tells when each test starts.But I think the fully general way to approach Who Tests What is to not assume that Who means "which test." There are other uses for this feature, so instead of hard-coding it to "test", I'm thinking in terms of the more general concept of "context." Often, the context would be "the current test," but maybe you're only interested in "Python version", or "subsystem," or "unit/integration/load."So the question is, how to know when contexts begin and end? Clearly with this general an idea, can't know. already has a plugin mechanism, so it seems like we should allow a plugin to determine the boundaries of contexts. can provide a plugin implementation that suffices for most people.A context will be a string, and each different context will have its own collected coverage data. In the discussion on issue 170, you can see people suggesting that we collect an entire stack trace for each line executed. This seems to me to be enormously more bulky to collect, more difficult to make use of, and ultimately not as flexible as simply noting a string context.There might be interesting things you can glean from that compendium of stack traces. I'd like to hear from you if you have ideas of things to do with stack traces that you can't do with contexts.Another minor point: what should be done with code executed before any context is established? I guess an None context would be good enough.Storing dataHaving multiple contexts will multiply the amount of data to be stored. It's hard to guess how much more, since that will depend on how overlapping your contexts are. My crude first guess is that large projects would have roughly C/4 times more data, where C is the number of contexts. If you have 500 [...]

Mac un-installs


The Mac is a nice machine and operating system, but there's one part of the experience I don't understand: software installation and uninstallation. I'm sure the App Store is meant to solve some of this, but the current situation is oddly manual.

Usually when I install applications on the Mac, I get a .dmg file, I open it, and there's something to copy to the Applications folder. Often, the .dmg window that opens has a cute graphic as a background, to encourage me to drag the application to the folder.

Proponents of this say, "it's so simple! The whole app is just a folder, so you can just drag it to Applications, and you're done. When you don't want the application any more, you just drag the application to the Trash."

This is not true. Applications may start self-contained in a folder, but they write data to other places on the disk. Those places are orphaned when you discard the application. Why is there no uninstaller to clean up those things?

As an example, I was cleaning up my disk this morning. Grand Perspective helped me find some big stuff I didn't need. One thing it pointed out to me was in a Caches folder. I wondered how much stuff was in folders called Caches:

sudo find / -type d -name '*Cache*' -exec du -sk {} \; -prune 2>&-

(Find every directory with 'Cache' in its name, show its disk usage in Kb, and don't show any errors along the way.) This found all sorts of interesting things, including folders from applications I had long ago uninstalled.

Now I could search for other directories belonging to these long-gone applications. For example:

sudo find / -type d -name '*TweetDeck*' -exec du -sh {} \; -prune 2>&-
 12K    /Users/ned/Library/Application Support/Fluid/FluidApps/TweetDeck
 84K    /Users/ned/Library/Caches/com.fluidapp.FluidApp.TweetDeck
 26M    /Users/ned/Library/Containers/com.twitter.TweetDeck
1.7M    /Users/ned/Library/Saved Application State/com.fluidapp.FluidApp.TweetDeck.savedState
sudo find / -type d -name '*twitter-mac*' -exec du -sh {} \; -prune 2>&-
288K    /private/var/folders/j2/gr3cj3jn63s5q8g3bjvw57hm0000gp/C/com.twitter.twitter-mac
 99M    /Users/ned/Library/Containers/com.twitter.twitter-mac
4.0K    /Users/ned/Library/Group Containers/

That's about 128Mb of junk left behind by two applications I no longer have. In the scheme of things, 128Mb isn't that much, but it's a lot more disk space than I want to devote to applications I've discarded. And what about other apps I tried and removed? Why leave this? Am I missing something that should have handled this for me?

One of Them


I have not written here about this year's presidential election. I am as startled, confused, and dismayed as many others about how Trump has managed to con people into following him, with nothing more than bluster and lies.It feels enormous to take it on in writing. Nathan Uno also feels as I do, but for different reasons. I've never met Nathan: he's an online friend, part of a small close-knit group who mostly share a religious background, and who enjoy polite intellectual discussions of all sorts of topics. I'm not sure why they let me in the group... :)Nathan and I started talking about our feelings about the election, and it quickly became clear that he had a much more visceral reason to oppose Trump than I did. I encouraged him to write about it, and he did. Here it is, "One of Them."•    •    •One of ThemArmed police came in the middle of the night and in the middle of winter, to take a husband away from his wife and a father away from his children. No explanation was given and his family was not allowed to see him or even know where he was being held. A few months later the man’s wife and children were also rounded up and taken away. They had only the belongings that they could carry with them, leaving everything else to be lost or stolen or claimed by others, including some of the family’s most precious possessions. The family was imprisoned in a camp surrounded by barbed wire and armed soldiers. They had little food and little heat and absolutely no freedom. A few months after the wife and children arrived they were finally reunited with their husband and father, seven months after he was taken from them in the night. They remained together at the camp for years until being released, given $25 and a bus ticket each, and left to try to put their shattered lives back together.No member of the family was ever charged with a crime. In fact, no member of the family was ever even suspected of a crime. They were imprisoned, along with tens of thousands of others, simply for being “one of them.”This is the story of my grandfather’s family. And my grandmother’s family. And tens of thousands of other families of Japanese descent who had the misfortune of living on the Pacific coast of the United States after the attack on Pearl Harbor.In the 1980s the U.S. government formally apologized, acknowledging their mistake, and financial reparations were made. Growing up I believed that we, as a country, had moved on, had learned a lesson. It never occurred to me that such a thing could happen again. And yet here we are, with a presidential candidate who has openly advocated violence against his opponents and detractors, offered to pay legal fees for those who break the law on his behalf, recommended policies that would discriminate against people based on their ethnicity, religion, or country of ancestry, suggested that deliberately killing women and children might be an appropriate response to terrorism, and yes, even said that he “might have” supported the policies that imprisoned my family. Xenophobic public policy leaves enduring scars on our society, scars that may not be obvious at first. We have Chinatowns today largely because public policy in San Fransisco in the late 1800s pushed Chinese immigrants to a live in a specific neighborhood. The proliferation of Chinese restaurants and Chinese laundries in our country can be traced back to the same time period, when policy restricted employment opportunities for Chinese immigrants and pushed them into doing low-paying “women’s work," like cooking and cleaning.I’ve chosen to make my point with these simple examples from the history of Asian Americans because that’s my heritage. But these examples are trivial compared to the deep, ugly scars left on o[...]