Subscribe: Planet Python
http://www.planetpython.org/rss20.xml
Added By: Feedage Forager Feedage Grade B rated
Language: English
Tags:
admin  ayrton  code  data import  data  django  file  import tool  import  line  new  pep  pig  projects  python  tool 
Rate this Feed
Rate this feedRate this feedRate this feedRate this feedRate this feed
Rate this feed 1 starRate this feed 2 starRate this feed 3 starRate this feed 4 starRate this feed 5 star

Comments (0)

Feed Details and Statistics Feed Statistics
Preview: Planet Python

Planet Python



Planet Python - http://planetpython.org/



 



Django Weekly: Django Weekly 16 - Djangocon Europe, cookiecutter, caching, djangogirls and more ...

Thu, 08 Dec 2016 10:28:19 +0000

Worthy ReadPresenting DjangoCon Europe 20172017’s DjangoCon Europe takes place against the gorgeous backdrop of Florence in the springtime. The event will be held in the historic Odeon Cinema in the centre of the city. Ticket sales and call for proposal is open. djangocon, europeCompose - Cloud Hosted Databases9 DBs to choose from, 5 min setup, auto-scaling, Cloud hosted. FREE for 30 Days. sponsorExploring with CookiecutterCookiecutter is ” A command-line utility that creates projects from cookiecutters (project templates). E.g. Python package projects, jQuery plugin projects.” In particular I explored some of the Django-specific cookiecutter templates. List includes Cookiecutter-Django, Django-crud, Django-paas, Simple Django. cookiecutter, two_scoops_of_djangoView / Browse all Django Admin Recent ActionsBasic tutorial showing us where django admin's recent actions are stored i.e. django_admin_log / LogEntry. Followed up with how to browse all entries from admin panel itself. Also shows how to disable edits and delete. admin-panelDjango Software Foundation Public RecordsIf you are looking for minutes of meeting of Django Software Foundation this is page to check. DSFDjango Girls Introduces Women to Programming at Andela NairobiCurators Note - Django Girls is phenomenal in it's execution and impact. I was at PyLadies SFO in March 2016 and met 3 journalist that credited Django Girls for creating an environment that was welcoming of women and newbies. Subsequently at Pycon India I met four college students who credited Django Girls for helping get internships. If you look at the no of cities and grassroot level events they do it's amazing. djangogirlsNew Relic InfrastructureMove fast, with confidence. Learn more about Infrastructure at an upcoming webinar. sponsorCache Me If You CanBackground and Explanation of a Django Caching Decorator CacheHow we used generic relations to add comments to model instancesIn a simple way we can say that Generic Relations is a foreign key that can refer to any instance of models. Good beginners tutorial explaining Generic Relations. ormPackaging static Sphinx-based documentation in a reusable Django app - By Albert Hopkins documentationA streamable "export to CSV" admin actionCode snippet code_snippetBuilding, Testing and Deploying Django App with Bitbucket PipelinesContinuous delivery with Bitbucket Pipelines and custom CentOS7 PostgreSQL Docker image. cookiecutter, continous delivery Projectsdjango-search-admin-autocomplete - 8 Stars, 1 ForkSimple django app that add autocomplete to search inside admin panel.[...]



hypothesis.works articles: Compositional shrinking

Thu, 08 Dec 2016 09:00:00 +0000

In my last article about shrinking, I discussed the problems with basing shrinking on the type of the values to be shrunk.

In writing it though I forgot that there was a halfway house which is also somewhat bad (but significantly less so) that you see in a couple of implementations.

This is when the shrinking is not type based, but still follows the classic shrinking API that takes a value and returns a lazy list of shrinks of that value. Examples of libraries that do this are theft and QuickTheories.

This works reasonably well and solves the major problems with type directed shrinking, but it’s still somewhat fragile and importantly does not compose nearly as well as the approaches that Hypothesis or test.check take.

Ideally, as well as not being based on the types of the values being generated, shrinking should not be based on the actual values generated at all.

This may seem counter-intuitive, but it actually works pretty well.

Read more...



Swisscom ICT: Smart Contracts

Thu, 08 Dec 2016 06:00:01 +0000

Paying is easy and seamless in Switzerland. And thanks to the public land register, even real estate transactions carry virtually no risk of fraud.  Contrast this with other countries, where you have to obtain insurance against an unpaid mortgage by a previous owner being discovered on the house you just bought. Or simply transferring money in rural Africa, where the risk of theft was high before the advent of mobile payment services. So there’s an obvious need for efficient transaction solutions (like Monetas) in markets that lack an efficient infrastructure. But what would be the benefit in a country like Switzerland, where the population is also reluctant to try new things that don’t have an obvious benefit? I believe that a very important element of the transaction landscape is the Smart Contract. The Smart Contract is basically an agreement that is defined by executable code instead of natural language text. So a mortgage contract would actually effect the interest payment instead of describing when it is due; a rental car contract would unlock the car instead of describing when you can use it.   The crypto transaction landscape Quite a few solutions and frameworks have been proposed that make use of cryptographic technology to ease transactions. Most notably are blockchains, which form the basis of public ecosystems like Bitcoin or Ethereum and restricted access frameworks like Hyperledger. Smart Contracts rely on a blockchain to add code in a non-disputable transaction. They do not necessarily involve a currency like Bitcoin, but may use one or more such currencies or other payment systems during the execution of a contract. Interactive contracts: Queries and Simulation Smart contracts are defined by executable code. They will need a human-readable high-level programming language and suitable execution environment. I fear this will be a custom language or JavaScript and I hope it will rather be Python [see inset below]. A very high level of abstraction is necessary: There will be contracts built upon other contracts, contracts reacting to input events. Contracts may provide a query API, answering questions about possible behavior (Was this agreed? What did we agree?). Contract conditions can also be simulated to check how contracts react. This may be used to discover gaps and expose potential risks. Will contracts ever negotiate with a mediator bot appointed by their parties to close discovered gaps? Insurance While reading the description of a structured financial product may seem complicated, smart contracts may cause an explosion of legal complexity. But their interactive nature and simulation possibilities help taming the complexity. Instead of paying lawyers fighting for their interpretation of the contract’s text, you might simply buy insurance to protect you against contractual risks. And the regulation, instead of issuing more complex regulations, might mandate that contract holders are insured, much in the same way that car owners have to be insured. The insurance will of course issue a smart contract, accessing the contracts it protects, along with other information about your life. Breaking the chain of transparency If an insurance company needs to see all my contracts and certain other life conditions, will this lead to a horror world of full transparency, as depicted by Dave Eggers in The Circle? Not necessarily: Simulation can be done without my identity, by someone I trust. The insurance doesn’t need to know me, as long as it gets the aggregated, anonymized data. But this needs a network of trust, similar to the banking network. Essential to this is an open way to express smart contracts. Open in the sense of open source, so everyone can inspect and vet the contractual building blocks. Technical Opinion: Why Python for smart contracts A full language with a broad community has advantages over a more restricted domain-specific language such as Ethereum’s  Solidity, which is JavaScript-like. Python is an expressive language wit[...]



Calvin Spealman: Finding "One Game A Month"

Thu, 08 Dec 2016 01:25:12 +0000

I was really excited about the One Game A Month challenge as soon as I heard about it.

For about two years I've struggled in fits and starts to make my way into game development. This hasn't been productive in any of the ways I hoped when I started. Its really difficult to be fairly experienced as a developer, which I believe I am in my day job as a web developer, while struggling really hard at an area in which your experience just doesn't exist.

Its like being a pilot who doesn't know how to drive.


But this challenge provided a new breath to this little hobby of mine. It gave me a scaffolding to experiment, to learn, to reflect on finished projects. I had spent far too much time on game projects that stretched on far past their exciting phases, bogged down by bad decisions and regret.

And it has worked.

I have a lot to learn. I have a lot of experience to gain through trial and error and mistake and discovery. I have a lot of fun to be had making more small games that scratch the little itches that have been building up. I have a really exciting 2017 planned for game development!

I've been doing One Game A Month for three months now. I've released three small games. These aren't great games by any stretch of the definition, of course. But they're games that I'm learning from. They're something I'm really excited about and I have needed something personally exciting as a developer for a long time. These are good times for me, and I hope to have more to share about them soon.



Daniel Bader: Cool new features in Python 3.6

Thu, 08 Dec 2016 00:00:00 +0000

Cool new features in Python 3.6 Python 3.6 adds a couple of new features and improvements that’ll affect the day to day work of Python coders. In this article I’ll give you an overview of the new features I found the most interesting. Improved numeric literals This is a syntactic tweak that makes numeric literals easier to read. You can now add underscores to numbers in order to group them to your liking. This is handy for expressing large quantities or constants in binary or hexadecimal: >>> six_figures = 100_000 >>> six_figures 100000 >>> programmer_error = 0xbad_c0ffee >>> flags = 0b_0111_0101_0001_0101 Remember, this change doesn’t introduce any new semantics. It’s just a way to represent numeric literals differently in your source code. A small but neat addition. You can learn more about this change in PEP 515. String interpolation Python 3.6 adds yet another way to format strings called Formatted String Literals. This new way of formatting strings lets you use embedded Python expressions inside string constants. Here’s are two simple examples to give you a feel for the feature: >>> name = 'Bob' >>> f'Hello, {name}!' 'Hello, Bob!' >>> a = 5 >>> b = 10 >>> f'Five plus ten is {a + b} and not {2 * (a + b)}.' 'Five plus ten is 15 and not 30.' String literals also support the existing format string syntax of the str.format() method. That allows you to do things like: >>> error = 50159747054 >>> f'Programmer Error: {error:#x}' 'Programmer Error: 0xbadc0ffee' Python’s new Formatted String Literals are similar to the JavaScript Template Literals added in ES2015/ES6. I think they’re quite a nice addition to the language and I look forward to using them in my day to day work. You can learn more about this change in PEP 498. Type annotations for variables Starting with Python 3.5 you could add type annotations to functions and methods: >>> def my_add(a: int, b: int) -> int: >>> return a + b In Python 3.6 you can use a syntax similar to type annotations for function arguments to type-hint standalone variables: >>> python_version : float = 3.6 Nothing has changed in terms of the semantics—CPython simply records the type as a type annotation but doesn’t validate or check types in any way. Type-checking is purely optional and you’ll need a tool like Mypy for that, which basically works like a code linter. You can learn more about this change in PEP 526. Other notable changes I think Python 3.6 will be an interesting release. There are many more interesting additions and improvements that are worth checking out. You can learn more about them in the links below or by reading the official “What’s New In Python 3.6” announcement. Syntax for asynchronous generators Syntax for asynchronous comprehensions Dictionaries are faster and use 20% to 25% less memory The typing module is considered a stable API A new “secrets” module for generating cryptographically secure authentication tokens …and more [...]



Machinalis: PyConAr 2016

Wed, 07 Dec 2016 20:02:03 +0000

A few days ago (November 25, 26 and 27th) the 8th edition of PyCon Argentina took place in Bahía Blanca, a city 600km to the south of Buenos Aires, and I was lucky enough to be able to go with a group of other people from Machinalis. PyConAr is the biggest Python related conference we have in Argentina, and one of the biggest programming conferences in the country too. It’s organized by Python Argentina (PyAr), a large community of programmers from all kind of backgrounds, which also holds smaller PyDays and one PyCamp each year. And quite important too: each year it’s organized in a different city by a different group of people, because as a community we want to give everyone the chance to participate, especially those who live far away from the usual places where conferences are held. The talks and workshops The topics were diverse, ranging from machine learning to asyncio, web APIs, mobile development, remote working, docker, and much more. And while most of the talks and workshops were very technical, I was glad to find some non-technical ones too. Specially the keynotes, which weren’t python-specific but dealt with topics of central interest in our community. Some quite subjective highlights: I loved the deep learning workshop organized by LinuxChix Argentina and guided by Celia Cintas. The example used in the workshop was Star Wars spaceships classification using neural networks and real time video. How awesome is that! [photo by Cynthia Monastirsky] I was happy about how our own 3 talks combined each other to give a general overview of the Python data science ecosystem. Rafael Carrascosa presented a general overview of the data science toolbelt, Javier Mansilla dived a little deeper into the deep learning scenario (pun intended), and I covered technical details of neural networks and Keras (the talks were filmed by the conference and will be uploaded, we will add the links later on). [photo by Juan Pedro Fisanotti] The sprints were quite productive! People helped solve issues and implemented new features into three not so known but very useful projects Pysenteishon, SimpleAI, and Twistorpy. [photo by PyConAr] I learned some interesting things regarding high speed access to large volumes of shared data from Claudio Freire’s talk. And finally, there was a workshop for blind people which I wasn’t able to be in, but others spoke of how useful it was for the participants! The community PyAr is known to be a very friendly and welcoming community. As we like to say, “PyAr es amor” (PyAr is love). This PyConAr wasn’t the exception to the rule. Experienced people guiding new ones in the workshops and sprints. Large groups of people who didn’t know each other before, enjoying a beer or a pizza after the conference with the mandatory geeky conversations (yes, we tried to define monads). Knowledge and smiles shared in the halls. I really enjoy being part of this big family that is PyAr :) Maybe one of the most emotive moments was the lightning talk given by one of the youngest participants of the workshop for blind people, in which she expressed how important it was for her. As one of the organizers said, "that alone is worth all the effort". [photo by Yamila Cuestas] Thanks For that and all the other things not included in this post, we would like to thank specially to this year organizers! You did an awesome job, and the whole PyAr community owes you their gratitude. And also to the sponsors, speakers, and everyone who helped in some way. See you in the next PyConAr! (or PyDay, or PyCamp :) The Machi-group :) [photo by Nicolás Demarchi] [...]



Enthought: Using the Canopy Data Import Tool to Speed Cleaning and Transformation of Data & New Release Features

Wed, 07 Dec 2016 19:59:04 +0000

Download Canopy to try the Data Import Tool In November 2016, we released Version 1.0.6 of the Data Import Tool (DIT), an addition to the Canopy data analysis environment. With the Data Import Tool, you can quickly import structured data files as Pandas DataFrames, clean and manipulate the data using a graphical interface, and create reusable Python scripts to speed future data wrangling. For example, the Data Import Tool lets you delete rows and columns containing Null values or replace the Null values in the DataFrame with a specific value. It also allows you to create new columns from existing ones. All operations are logged and are reversible in the Data Import Tool so you can experiment with various workflows with safeguards against errors or forgetting steps. What’s New in the Data Import Tool November 2016 Release Pandas 0.19 support, re-usable templates for data munging, and more. Over the last couple of releases, we added a number of new features and enhanced a number of existing ones. A few notable changes are: The Data Import Tool now supports the recently released Pandas version 0.19.0. With this update, the Tool now supports Pandas versions 0.16 through 0.19. The Data Import Tool now allows you to delete empty columns in the DataFrame, similar to existing option to delete empty rows. The Data Import Tool allows you to choose how to delete rows or columns containing Null values: “Any” or “All” methods are available. The Data Import Tool automatically generates a corresponding Python script for data manipulations performed in the GUI and saves it in your home directory re-use in future data wrangling. Every time you successfully import a DataFrame, the Data Import Tool automatically saves a generated Python script in your home directory. This way, you can easily review and reproduce your earlier work. The Data Import Tool generates a Template with every successful import. A Template is a file that contains all of the commands or actions you performed on the DataFrame and a unique Template file is generated for every unique data file. With this feature, when you load a data file, if a Template file exists corresponding to the data file, the Data Import Tool will automatically perform the operations you performed the last time. This way, you can save progress on a data file and resume your work. Along with the feature additions discussed above, based on continued user feedback, we implemented a number of UI/UX improvements and bug fixes in this release. For a complete list of changes introduced in Version 1.0.6 of the Data Import Tool, please refer to the Release Notes page in the Tool’s documentation.     Example Use Case: Using the Data Import Tool to Speed Data Cleaning and Transformation Now let’s take a look at how the Data Import Tool can be used to speed up the process of cleaning up and transforming data sets. As an example data set, let’s take a look at the Employee Compensation data from the city of San Francisco. NOTE: You can follow the example step-by-step by downloading Canopy and starting a free 7 day trial of the data import tool Step 1: Load data into the Data Import Tool First we’ll download the data as a .csv file from the San Francisco Government data website, then open it from File -> Import Data -> From File… menu item in the Canopy Editor (see screenshot at right). After loading the file, you should see the DataFrame below in the Data Import Tool: The Data Import Tool automatically detects and converts column types (in this case to an integer type). As you can see at the right, the Data Import Tool automatically detected and converted the columns “Job Code”, “Job Family Code” and “Union Code” to an Integer column type. But, if the Tool inferred erroneously, you can simply remove a specific column conversion by deleting it from the Edit Command window or [...]



Will McGugan: PyFilesystem 2.0 Released

Wed, 07 Dec 2016 17:58:25 +0000

I'd like to announce version 2.0.0 of PyFilesystem, which is now available on PyPi. PyFilesystem is a Python module I started some time in 2008, and since then it has been very much a part of my personal standard library. I've used it in personal and professional projects, as have many other developers and organisations. Recap If you aren't familiar with PyFilesystem; it's an abstraction layer for filesystems. Essentially anything with files and directories (hard-drive, zip file, ftp server, network filesystems etc.) may be wrapped with a common interface. With it, you can write code that is agnostic as to where the files are physically located. Here's a quick example that recursively counts the lines of code in a directory: def count_python_loc(fs): """Count non-blank lines of Python code.""" count = 0 for path in fs.walk.files(filter=['*.py']): with fs.open(path) as python_file: count += sum(1 for line in python_file if line.strip()) return count from fs import open_fs projects_fs = open_fs('~/projects') print(count_python_loc(projects_fs)) The fs argument to count_python_loc is an FS object, which encapsulates everything you would need to do with a filesystem. Because of this abstraction, the same code will work with any filesystem. For instance, counting the lines of code in a zip file is a single line change: projects_fs = open_fs('zip://projects.zip') See my previous posts on PyFilesystem for more back-story. © 2016 Will McGugan The tree method renders the filesystem structure with unicode box drawing characters. This can be a nice way of reporting file changes in a command line app, and a useful debugging aid in general. The fact that there are trees on my wallpaper is entirely coincidental. Why the update? A lot has happened since 2008. Python 3 happened. The IO library happened. Scandir happened. And while PyFilesystem has kept up with those developments, the code has suffered from the weight of small incremental changes. Not to the degree that it required a re-write perhaps, but a new version gave me the opportunity to make improvements to the API which couldn't be done without breaking a lot of code. The re-write was guided by 8 years of observations regarding what developers wanted from the library; what worked well, what felt awkward, and the numerous edge cases in creating the illusion that all filesystems work alike (they really don't). A lot of agonizing has gone in to the design of the new API to make it simple to use without sacrificing functionality. This often required breaking up large methods in to more atomic methods that do one thing and do it well. Another motivation for this version was to make it easier to implement new filesystems. It turns out that all you need to implement any filesystem is to write 7 methods. Which I find somewhat remarkable. The new API has been designed to make it much easier to develop your own filesystems in general, with more of the heavy lifting been done by the base class. I'm hoping this will encourage more developers to release new filesystems, and for pre-2.0.0 filesystems to be ported (which I'll gladly assist with). So what is new? Cleaner code The new project is a unified Python 2 and 3 code-base. It's legacy free code with 100% coverage. I wanted it to be bullet-proof because PyFilesystem an integral part of Moya. Moya's static server uses PyFilesystem to serve assets. Here's a screenshot of that in action: © 2016 Will McGugan Moya uses PyFilesystem in its static server. This screenshot shows what happens when you statically serve a FTPFS. Moving and copying got simpler. Here's how you compress your projects directory as a zip file: from fs.copy import copy_fs copy_fs("'~/projects", "'zip://~/projects.zip") This works because the fs.copy and fs.move modules accept both FS objects and FS URLs. Simple file [...]



Django Weekly: View / Browse all Django Admin Recent Actions

Wed, 07 Dec 2016 17:01:03 +0000

If you have used Django Admin Panel the below screenshot will look familiar.


(image)

On the right side you can see a listing of latest actions aka create, update, delete actions taken by the user. Wouldn't it be convenient to browse all the actions taken by users ?.

Where is this information stored ?.

Execute python manage.py dbshell you will be inside your database cli client, with the project's default database selected.
Execute select * from django_admin_log; ( I am using sqlite3 for the tutorial ).

sqlite> select * from django_admin_log;
1|1|CBSE|1|[{"added": {}}]|9|1|2016-12-07 15:23:39.916789
2|2|ICSE|1|[{"added": {}}]|9|1|2016-12-07 15:23:43.296926
3|3|IB|1|[{"added": {}}]|9|1|2016-12-07 15:23:49.387019
sqlite> 

django_admin_log is created as part of django admin panel models. Check contrib/admin/models.py. In class LogEntry check class Meta you can see db_table = 'django_admin_log'. So in django_admin_log table all entries reside.

Let us add the model to our admin panel for browsing all these entries.
Go inside your admin.py and add

from django.contrib import admin
from django.contrib.admin.models import LogEntry
admin.site.register(LogEntry)

You will now be able to browse all user actions.

(image)

However all entries are editable. Let us create a non-editable version.

class LogEntryAdmin(admin.ModelAdmin):
    readonly_fields = ('content_type',
        'user',
        'action_time',
        'object_id',
        'object_repr',
        'action_flag',
        'change_message'
    )

    def has_delete_permission(self, request, obj=None):
        return False

    def get_actions(self, request):
        actions = super(LogEntryAdmin, self).get_actions(request)
        del actions['delete_selected']
        return actions

ModelAdmin helps us override the default behaviour/UI of a model in context of the admin panel. First we have marked all fields as read only by adding them in readonly_fields. Django admin will not allow editing of these fields/columns. We over-ride the has_delete_permission method to return False, thus disabling the delete button at the bottom of the Details View page.

(image)

On the List View / where all entries are listed we have a dropdown allows users to delete any selected object/entry. We over-ride the `get_actions` method and remove the delete_selected actions/method from the actions. Now one can browse all the log entries but cannot delete them.

(image)



PyCharm: PyCharm 2016.3.1 RC Available

Wed, 07 Dec 2016 15:25:33 +0000

We’re happy to announce the availability of PyCharm 2016.3.1 RC. We’ve worked hard to fix some issues some of you are facing:

  • If you would like to disable the automatic activation of the project’s virtualenv, you can now disable this setting in Settings | Tools | Terminal | Activate virtualenv
  • Terminal path can be configured globally
  • The terminal on macOS will source your bash_profile upon activation
  • HTML tags automatically close in Django templates
  • Python Console: Execute code in console (Shift+Alt+E) indentation fixed, tab completion
  • And various other bugs, see the release notes for details

Get it now from the EAP page!

Although we pay careful attention to make sure our software works well, if you experience any issues please report them on our issue tracker. If you have any ideas about how we can make PyCharm better, please let us know on our issue tracker as well!

The Drive to Develop

-PyCharm Team

(image)



Codementor: 15 Essential Python Interview Questions

Wed, 07 Dec 2016 14:49:41 +0000

##Introduction Looking for a Python job? Chances are you will need to prove that you know how to work with Python. Here are a couple of questions that cover a wide base of skills associated with Python. Focus is placed on the language itself, and not any particular package or framework. Each question will be linked to a suitable tutorial if there is one. Some questions will wrap up multiple topics. I haven’t actually been given an interview test quite as hard as this one, if you can get to the answers comfortably then go get yourself a job. ##What this tutorial is not This tutorial does not aim to cover every available workplace culture - different employers will ask you different questions in different ways; they will follow different conventions; they will value different things. They will test you in different ways. Some employers will sit you down in from of a computer and ask you to solve simple problems; some will stand you up in front of a white board and do similar; some will give you a take home test to solve; some will just have a conversation with you. The best test for a programmer is actually programming. This is a difficult thing to test with a simple tutorial. So for bonus points make sure that you can actually use the functionality demonstrated in the questions. If you actually understand how to get to the answers well enough that you can actually make use of the demonstrated concepts then you are winning. Similarly, the best test for a software engineer is actually engineering. This tutorial is about Python as a language. Being able to design efficient, effective, maintainable class hierarchies for solving niche problems is great and wonderful and a skill set worth pursuing but well beyond the scope of this text. Another thing this tutorial is not is PEP8 compliant. This is intentional as, as mentioned before, different employers will follow different conventions. You will need to adapt to fit the culture of the workplace. Because practicality beats purity. Another thing this tutorial isn’t is concise. I don’t want to just throw questions and answers at you and hope something sticks. I want you to get it, or at least get it well enough that you are in a position to look for further explanations yourself for any problem topics. Want to ace your technical interview? Schedule a Technical Interview Practice Session with an expert now! Question 1 What is Python really? You can (and are encouraged) make comparisons to other technologies in your answer ###Answer Here are a few key points: - Python is an interpreted language. That means that, unlike languages like C and its variants, Python does not need to be compiled before it is run. Other interpreted languages include PHP and Ruby. Python is dynamically typed, this means that you don’t need to state the types of variables when you declare them or anything like that. You can do things like x=111 and then x="I'm a string" without error Python is well suited to object orientated programming in that it allows the definition of classes along with composition and inheritance. Python does not have access specifiers (like C++’s public, private), the justification for this point is given as “we are all adults here” in Python, functions are first-class objects. This means that they can be assigned to variables, returned from other functions and passed into functions. Classes are also first class objects Writing Python code is quick but running it is often slower than compiled languages. Fortunately, Python allows the inclusion of C based extensions so bottlenecks can be optimised away and often are. The numpy package is a good example of this, it’s really quite quick because a lot of the number crunching it does isn’t actua[...]



tryexceptpass: I understand what you’re saying and agree that it’s a safer option than traditional threading, but…

Wed, 07 Dec 2016 13:55:18 +0000

There are quite a few thread-safe methods available throughout the library, some that were not discussed here, created specifically for…




Marcos Dione: ayrton-0.9.1

Wed, 07 Dec 2016 13:14:08 +0000

Last night I realized the first point. Checking today I found the latter. Early, often, go!

  • ayrton-0.9 has debug on. It will leave lots of files laying around your file system.
  • Modify the release script to do not allow this never ever more.
  • make install was not running the tests.

Get it on github or pypi!


python ayrton




Codementor: Adding Flow Control to Apache Pig using Python

Wed, 07 Dec 2016 09:38:58 +0000

(image source) ##Introduction So you like Pig but its cramping your style? Are you not sure what Pig is about? Are you keen to write some code to write code for you? If yes, then this is for you. This tutorial ties together a whole lot of different techniques and technologies. The aim here is to show you a trick to get Pig to behave in a way that’s just a little bit more loopy. It’s a trick I’ve used before quite a lot and I’ve written a couple of utility functions to make it easy. I’ll go over the bits and pieces here. This tutorial, on a more general note, is about writing code that writes code. The general technique and concerns outlined here can be applied to other code generating problems. ##What Does Pig Do? Pig is a high-level scripting toolset used for defining and executing complex map-reduce workflows. Let’s take a closer look at that sentence… Pig, is a top-level Apache project. It is open source and really quite nifty. Learn more about it here. PigLatin is Pig’s language. Pig executes PigLatin scripts. Within a PigLatin script you write a bunch of statements that get converted into a bunch of map-reduce jobs that can get executed in sequence on your Hadoop cluster. It’s usually nice to abstract away from writing plain old map-reduce jobs because they can be a total pain in the neck. If you haven’t used Pig before and aren’t sure if it’s for you, it might be a good idea to check out Hive. Hive and Pig have a lot of overlap in terms of functionality, but have different philosophies. They aren’t total competitors because they are often used in conjunction with one another. Hive resembles SQL, while PigLatin resembles… PigLatin. So if you are familiar with SQL then Hive might be an easier learn, but IMHO Pig is a bit more sensible than Hive in how it describes data flow. ##What Doesn’t Pig Do? Pig doesn’t make any decisions about the flow of program execution, it only allows you to specify the flow of data. In other words, it allows you to say stuff like this: ----------------------------------------------- -- define some data format goodies ----------------------------------------------- define CSV_READER org.apache.pig.piggybank.storage.CSVExcelStorage( ',', 'YES_MULTILINE', 'UNIX' ); define CSV_WRITER org.apache.pig.piggybank.storage.CSVExcelStorage( ',', 'YES_MULTILINE', 'UNIX', 'SKIP_OUTPUT_HEADER' ); ----------------------------------------------- -- load some data ----------------------------------------------- r_one = LOAD 'one.csv' using CSV_READER AS (a:chararray,b:chararray,c:chararray); r_two = LOAD 'two.csv' using CSV_READER AS (a:chararray,d:chararray,e:chararray); ----------------------------------------------- -- do some processing ----------------------------------------------- r_joined = JOIN r_one by a, t_two by a; r_final = FOREACH r_joined GENERATE r_one::a, b, e; ----------------------------------------------- -- store the result ----------------------------------------------- store r_final into 'r_three.csv' using CSV_WRITER; The script above says where the data should flow. Every statement you see there will get executed exactly once no matter what (unles[...]



Codementor: Extending Apache Pig with Python UDFs

Wed, 07 Dec 2016 09:31:31 +0000

(image source) Introduction Apache Pig is a popular system for executing complex Hadoop map-reduce based data-flows. It adds a layer of abstraction on top of Hadoop’s map-reduce mechanisms in order to allow developers to take a high-level view of the data and operations on that data. Pig allows you to do things more explicitly. For example, you can join two or more data sources (much like an SQL join). Writing a join as a map and reduce function is a bit of a drag and it’s usually worth avoiding. So Pig is great because it simplifies complex tasks - it provides a high-level scripting language that allows users to take more of a big-picture view of their data flow. Pig is especially great because it is extensible. This tutorial will focus on its extensibility. By the end of this tutorial, you will be able to write PigLatin scripts that execute Python code as a part of a larger map-reduce workflow. Pig can be extended with other languages too, but for now we’ll stick to Python. Before we continue This tutorial relies on a bunch of knowledge. It’ll be very useful if you know a little Python and PigLatin. It’ll also be useful to know a bit about how map-reduce works in the context of Hadoop. User Defined Functions (UDFs) A Pig UDF is a function that is accessible to Pig, but written in a language that isn’t PigLatin. Pig allows you to register UDFs for use within a PigLatin script. A UDF needs to fit a specific prototype - you can’t just write your function however you want because then Pig won’t know how to call your function, it won’t know what kinds of arguments it needs, and it won’t know what kind of return value to expect. There are a couple of basic UDF types: Eval UDFs This is the most common type of UDF. It’s used in FOREACH type statements. Here’s an example of an eval function in action: users = LOAD 'user_data' AS (name: chararray); upper_users = FOREACH users GENERATE my_udfs.to_upper_case(name); This code is fairly simple - Pig doesn’t really do string processing so we introduce a UDF that does. There are some missing pieces that I’ll get to later, specifically how Pig knows what my_udfs means and suchlike. Aggregation UDFs These are just a special case of an eval UDF. An Aggregate function is usually applied to grouped data. For example: user_sales = LOAD 'user_sales' AS (name: chararray, price: float); grouped_sales = GROUP user_sales BY name; number_of_sales = FOREACH grouped_sales GENERATE group, COUNT(user_sales); In other words, an aggregate UDF is a udf that is used to combine multiple pieces of information. Here we are aggregating sales data to show how many purchases were made by each user. Filter UDFs A filter UDF returns a boolean value. If you have a data source that has a bunch of rows and only a portion of those rows are useful for the current analysis then a filter function of some kind would be useful. An example of a filter function is action follows: user_messages = LOAD 'user_twits' AS (name:chararray, message:chararray); rude_messages = FILTER user_messages by my_udfs.contains_naughty_words(message); Enough talk, let’s code In this section we’ll be writing a couple of Python UDFs and making them accessible within PigLatin scripts. Here’s about the simplest Python UDF you can write: from pig_util import outputSchema @outputSchema('word:chararray') def hi_world(): return "hello world" The data output from a function has a specific form. Pig likes it if you specify the schema of the data because then it knows what it can do with that data. That’s what the output_schema decorator is for. There are a bunch of different ways to specify a schema, we’ll get to[...]



Python Insider: Python 3.6.0 release candidate is now available

Wed, 07 Dec 2016 02:34:29 +0000

Python 3.6.0rc1 is the release candidate for Python 3.6, the next major release of Python. Code for 3.6.0 is now frozen.  Assuming no release critical problems are found prior to the 3.6.0 final release date, currently 2016-12-16, the3.6.0 final release will be the same code base as this 3.6.0rc1.Maintenance releases for the 3.6 series will follow at regularintervals starting in the first quarter of 2017. Among the major new features in Python 3.6 are: * PEP 468 - Preserving the order of **kwargs in a function * PEP 487 - Simpler customization of class creation * PEP 495 - Local Time Disambiguation * PEP 498 - Literal String Formatting * PEP 506 - Adding A Secrets Module To The Standard Library * PEP 509 - Add a private version to dict * PEP 515 - Underscores in Numeric Literals * PEP 519 - Adding a file system path protocol * PEP 520 - Preserving Class Attribute Definition Order * PEP 523 - Adding a frame evaluation API to CPython * PEP 524 - Make os.urandom() blocking on Linux (during system startup) * PEP 525 - Asynchronous Generators (provisional) * PEP 526 - Syntax for Variable Annotations (provisional) * PEP 528 - Change Windows console encoding to UTF-8 * PEP 529 - Change Windows filesystem encoding to UTF-8 * PEP 530 - Asynchronous Comprehensions Please see "What’s New In Python 3.6" for more information: https://docs.python.org/3.6/whatsnew/3.6.html You can find Python 3.6.0rc1 here: https://www.python.org/downloads/release/python-360rc1/ Note that 3.6.0rc1 is still a preview release and thus its use is not recommended for production environments More information about the release schedule can be found here: https://www.python.org/dev/peps/pep-0494/ [...]



Catalin George Festila: The python-nmap python module fail.

Tue, 06 Dec 2016 22:15:46 +0000

You can read about this python module here.

First let's install this python module.
C:\Python27>cd Scripts

C:\Python27\Scripts>pip install python-nmap
Collecting python-nmap
Downloading python-nmap-0.6.1.tar.gz (41kB)
100% |################################| 51kB 240kB/s
Installing collected packages: python-nmap
Running setup.py install for python-nmap ... done
Successfully installed python-nmap-0.6.1

About this python-nmap version you can read here.
I try to run the example source code but not of this example working.
For example I got this:
>>> nm.scan('127.0.0.1', '22-443')
{'nmap': {'scanstats': {'uphosts': '1', 'timestr': 'Wed Dec 07 08:13:01 2016', '
downhosts': '-1', 'totalhosts': '0', 'elapsed': '10.74'}, 'scaninfo': {'tcp': {'
services': '22-443', 'method': 'syn'}, 'error': [u'dnet: Failed to open device l
o0\r\nQUITTING!\r\n', u'dnet: Failed to open device lo0\r\nQUITTING!\r\n']}, 'co
mmand_line': 'nmap -oX - -p 22-443 -sV 127.0.0.1'}, 'scan': {}}






Brett Cannon: What to look for in a new TV

Tue, 06 Dec 2016 19:13:23 +0000

I'm kind of an A/V nerd. Now I'm not hardcore enough to have a vinyl collection or have an amp for my TV, but all my headphones cost over $100 and I have a Sonos Playbar so I don't have to put up with crappy TV speakers. What I'm trying to say is that I care about the A/V equipment I use, but not to the extent that money is no object when it comes to my enjoyment of a movie (I'm not that rich and my wife would kill me if I spent that kind of money on electronics). That means I tend to research extensively before making a major A/V purchase since I don't do it very often and I want quality within reason which does not lend itself to impulse buying. Prior to September 1, 2016, I had a 2011 Vizio television. It was 47", did 1080p, and had passive 3D. When I purchased the TV I was fresh out of UBC having just finished my Ph.D. so it wasn't top-of-the-line, but it was considered very good for the price. I was happy with the picture, but admittedly it wasn't amazing; the screen had almost a matte finish which led to horrible glare. I also rarely used the 3D in the television as 3D Blu-Ray discs always cost extra and so few movies took the time to actually film in 3D to begin with, instead choosing to do it in post-production (basically animated films and TRON: Legacy were all that we ever watched in 3D). And to top it all off, the TV took a while to turn on. I don't know what kind of LCB bulbs were in it, but they took forever to warm up and just annoyed me (yes, very much a first-world problem). So when UHD came into existence I started to keep an eye on the technology and what television manufacturers were doing to incorporate the technology to entice people like me to upgrade. After two years of watching this space and one of the TVs I was considering having a one-day sale that knock 23% off the price, I ended up buying a 55" Samsung KS8000 yesterday. Since I spent so much time considering this purchase I figured I would try and distill what knowledge I have picked up over the years into a blog post so that when you decide to upgrade to UHD you don't have to start from zero knowledge like I did. What to care about First, you don't care about the resolution of the TV. All UHD televisions are 4K, so that's just taken care of for you. It also doesn't generally make a difference in the picture because most people sit too far away from their TV to make the higher resolution matter. No, the one thing you're going to care about is HDR and everything that comes with it. And of course it can't be a simple thing to measure like size or resolution. Oh no, HDR has a bunch of parts to it that go into the quality of the picture: brightness, colour gamut, and format (yes, there's a format war; HD-DVD/Blu-Ray didn't teach the TV manufacturers a big enough lesson). Brightness A key part of HDR is the range of brightness to show what you frequently hear referred to as "inky blacks" and "bright whites". The way you get deep blacks and bright whites is by supporting a huge range of brightness. What you will hear about TVs is what their maximum nit is. Basically you're aiming for 1000 nits or higher for a maximum and as close to 0 as possible for a minimum. Now of course this isn't as simple as it sounds as there's different technology being used to try and solve this problem. LCD Thanks to our computers I'm sure everyone reading this is familiar with LCD displays. But what you might not realize is how they exactly work. In a nutshell there are LED lightbulbs behind your screen that provides white light, and then the[...]



Brett Cannon: Introducing Which Film

Tue, 06 Dec 2016 19:13:23 +0000

What I'm announcing Today I'm happy to announce the public unveiling of Which Film! I'll discuss how the site came about and what drives it, but I thought I would first explain what it does: it's a website to help you choose what movie you and your family/friends should watch together. What you do is you go to the site, enter in the Trakt.tv usernames of everyone who wants to watch a film together (so you need at least two people and kept data like your watchlist and ratings on Trakt), and then Which Film cross-references everyone's watchlists and ratings to create a list of movies that people may want to watch together. The list of movies is ranked based on a simple point scale. If a movie is on someone's watchlist it gets 4 points, movies rated 10 ⭐ get 3 points, 9 ⭐ get 2 points, and 8 ⭐ get 1 point. Everyone who participates contributes points and the movies are sorted from highest score to lowest. The reason for the point values is the assumption that watching a movie most people have not seen is the best, followed by a movies people rate very highly. In the case of ties, the movie seen longest ago (if ever) by anyone in the group is ranked higher than movies seen more recently by someone. That way there's a bigger chance someone will be willing to watch a movie again when everyone else wants to see it for the first time. None of this is very fancy or revolutionary, but it's useful any time you get together with a group of friends to watch a film and you end up having a hard time choosing to watch. It can help even between spouses as it will identify movies both people want to watch, removing that particular point of contention. The story behind Which Film Now normally launching a new website wouldn't cause for any backstory, but this project has been under development for about six years, so there's a bit of history to it. One fateful night ... The inspiration for Which Film stemmed from one night when my co-creator Karl, his wife, my wife, and I got together and decided we wanted to watch a movie. This turned out to be quite an ordeal due to disparate tastes among all four of us. Karl and I thought that there had to be a better way to figure out a film we could all happily watch together. It didn't need to necessarily be something none of us had seen (although that was preferred), but it did need to be something that had a chance of making all of us happy if we chose to watch it. This is when I realized that at least for me I had all of the relevant data to make such a decision on IMDb. I had been keeping my watchlist and ratings up-to-date on the site for years, to the point of amassing a watchlist over of 400 movies. Karl and I realized that had all four of us done that we could have cross-referenced the data and easily have found a film we all liked. Yes, it would require convincing everyone involved to keep track of what movies they wanted to see and rating movies that had seen, but we figured that wasn't an insurmountable problem. And so we decided we should code up a solution since we're both software developers. You need an API, IMDb But there was trouble with this project from the beginning. It turns out that while IMDb is happy for you to store your data on their servers, they don't exactly make it easy to get the data out. For instance, when I started looking into this they had two ways of getting to your data in some programmatic way: RSS and CSV files. The problem with RSS is that it was capped at (I believe) 200 entries, so I couldn't use it to access my entire data set. The i[...]



Marcos Dione: ayrton-0.9

Tue, 06 Dec 2016 18:46:11 +0000

Another release, but this time not (only) a bugfix one. After playing with bool semantics I converted the file tests from a _X format, which, let's face it, was not pretty, into the more usual -X format. This alone merits a change in the minor version number. Also, _in, _out and _err also accept a tuple (path, flags), so you can specify things like os.O_APPEND. In other news, I had to drop support for Pyhton-3.3, because otherwise I would have to complexify the import system a lot. But in the end, yes, this also is a bugfix release. Lost of fd leaks where plugged, so I suggest you to upgrade if you can. Just remember the s/_X/-X/ change. I found all the leaks thanks to unitest's warnings, even if sometimes they were a little misleading: testRemoteCommandStdout (tests.test_remote.RealRemoteTests) ... ayrton/parser/pyparser/parser.py:175: ResourceWarning: unclosed self.stack[-1] = (dfa, next_state, node) The file and line cited in the warning have nothing to do with the warning itself (it was not the one who raised it) or the leaked fd, so it took me a while to find were those leaks were coming from. I hope I have some time to find why this is so. The most frustrating thing was that unitest closes the leaking fd, which is nice, but in one of the test cases it was closing it seemingly before the test finished, and the test failed because the socket was closed: ====================================================================== ERROR: testLocalVarToRemoteToLocal (tests.test_remote.RealRemoteTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/mdione/src/projects/ayrton_clean/ayrton/tests/test_remote.py", line 225, in wrapper test (self) File "/home/mdione/src/projects/ayrton_clean/ayrton/tests/test_remote.py", line 235, in testLocalVarToRemoteToLocal self.runner.run_file ('ayrton/tests/scripts/testLocalVarToRealRemoteToLocal.ay') File "/home/mdione/src/projects/ayrton_clean/ayrton/__init__.py", line 304, in run_file return self.run_script (script, file_name, argv, params) File "/home/mdione/src/projects/ayrton_clean/ayrton/__init__.py", line 323, in run_script return self.run_tree (tree, file_name, argv, params) File "/home/mdione/src/projects/ayrton_clean/ayrton/__init__.py", line 336, in run_tree return self.run_code (code, file_name, argv) File "/home/mdione/src/projects/ayrton_clean/ayrton/__init__.py", line 421, in run_code raise error File "/home/mdione/src/projects/ayrton_clean/ayrton/__init__.py", line 402, in run_code exec (code, self.globals, self.locals) File "ayrton/tests/scripts/testLocalVarToRealRemoteToLocal.ay", line 6, in with remote ('127.0.0.1', _test=True): File "/home/mdione/src/projects/ayrton_clean/ayrton/remote.py", line 362, in __enter__ i, o, e= self.prepare_connections (backchannel_port, command) File "/home/mdione/src/projects/ayrton_clean/ayrton/remote.py", line 270, in prepare_connections self.client.connect (self.hostname, *self.args, **self.kwargs) File "/usr/lib/python3/dist-packages/paramiko/client.py", line 338, in connect t.start_client() File "/usr/lib/python3/dist-packages/paramiko/transport.py", line 493, in start_client raise e File "/usr/lib/python3/dist-packages/paramiko/transport.py", line 1757, in run self.kex_engine.parse[...]



tryexceptpass: Threaded Asynchronous Magic and How to Wield It

Tue, 06 Dec 2016 18:06:16 +0000

Photo Credit: Daniel Schwen via WikipediaA dive into Python’s asyncio tasks and event loopsOk let’s face it. Clock speeds no longer govern the pace at which computer processors improve. Instead we see increased transistor density and higher core counts. Translating to software terms, this means that code won’t run faster, but more of it can run in parallel.Although making good use of our new-found silicon real estate requires improvements in software, a lot of programming languages have already started down this path by adding features that help with parallel execution. In fact, they’ve been there for years waiting for us to take advantage.So why don’t we? A good engineer always has an ear to the ground, listening for the latest trends in his industry, so let’s take a look at what Python is building for us.What do we have so far?Python enables parallelism through both the threading and the multiprocessing libraries. Yet it wasn’t until the 3.4 branch that it gave us the asyncio library to help with single-threaded concurrency. This addition was key in providing a more convincing final push to start swapping over from version 2.The asyncio package allows us to define coroutines. These are code blocks that have the ability of yielding execution to other blocks. They run inside an event loop which iterates through the scheduled tasks and executes them one by one. A task switch occurs when it reaches an await statement or when the current task completes.Task execution itself happens the same as in a single-threaded system. Meaning, this is not an implementation of parallelism, it’s actually closer to multithreading. We can perceive the concurrency in situations where a block of code depends on external actions.This illusion is possible because the block can yield execution while it waits, making anything that depends on external IO, like network or disk storage, a great candidate. When the IO completes, the coroutine receives an interrupt and can proceed with execution. In the meantime, other tasks execute.The asyncio event loop can also serve as a task scheduler. Both asynchronous and blocking functions can queue up their execution as needed.TasksA Task represents callable blocks of code designed for asynchronous execution within event loops. They execute single-threaded, but can run in parallel through loops on different threads.Prefixing a function definition with the async keyword turns it into an asynchronous coroutine. Though the task itself will not exist until it’s added to a loop. This is usually implicit when calling most loop methods, but asyncio.ensure_future(your_coroutine) is the more direct mechanism.To denote an operation or instruction that can yield execution, we use the await keyword. Although it’s only available within a coroutine block and causes a syntax error if used anywhere else.Please note that the async keyword was not implemented until Python version 3.5. So when working with older versions, use the @asyncio.coroutine decorator and yield from keywords instead.SchedulingIn order to execute a task, we need a reference to the event loop in which to run it. Using loop = asyncio.get_event_loop() gives us the current loop in our execution thread. Now it’s a matter of calling loop.run_until_complete(your_coroutine) or loop.run_forever() to have it do some work.Let’s look at a short example to illustrate a few points. I strongly encourage you to open an interpreter and follow along:import timeimport asyncioa[...]



Continuum Analytics News: Introducing: fastparquet

Tue, 06 Dec 2016 17:48:50 +0000

Developer Blog Tuesday, December 6, 2016 Martin Durant Continuum Analytics A compliant, flexible and speedy interface to Parquet format files for Python, fastparquet provides seamless translation between in-memory pandas DataFrames and on-disc storage. In this post, we will introduce the two functions that will most commonly be used within fastparquet, followed by a discussion of the current Big Data landscape, Python's place within it and details of how fastparquet fills one of the gaps on the way to building out a full end-to-end Big Data pipeline in Python. fastparquet Teaser New users of fastparquet will mainly use the functions write and ParquetFile.to_pandas. Both functions offer good performance with default values, and both have a number of options to improve performance further. import fastparquet # write data fastparquet.write('out.parq', df, compression='SNAPPY') # load data pfile = fastparquet.ParquetFile('out.parq') df2 = pfile.to_pandas() # all columns df3 = pfile.to_pandas(columns=['floats', 'times']) # pick some columns Introduction: Python and Big Data Python was named as a favourite tool for data science by 45% of data scientists in 2016. Many reasons can be presented for this, and near the top will be: Python is very commonly taught at college and university level Python and associated numerical libraries are free and open source The code tends to be concise, quick to write, and expressive An extremely rich ecosystem of libraries exist for not only numerical processing but also other important links in the pipeline from data ingest to visualization and distribution of results Big Data, however, has typically been based on traditional databases and, in latter years, the Hadoop ecosystem. Hadoop provides a distributed file-system, cluster resource management (YARN, Mesos) and a set of frameworks for processing data (map-reduce, pig, kafka, and many more). In the past few years, Spark has rapidly increased in usage, becoming a major force, even though 62% of users use Python to execute Spark jobs (via PySpark). The Hadoop ecosystem and its tools, including Spark, are heavily based around the Java Virtual Machine (JVM), which creates a gap between the familiar, rich Python data ecosystem and clustered Big Data with Hadoop. One such missing piece is a data format that can efficiently store large amounts of tabular data, in a columnar layout, and split it into blocks on a distributed file-system. Parquet has become the de-facto standard file format for tabular data in Spark, Impala and other clustered frameworks. Parquet provides several advantages relevant to Big Data processing: Columnar storage, only read the data of interest Efficient binary packing Choice of compression algorithms and encoding Splits data into files, allowing for parallel processing Range of logical types Statistics stored in metadata to allow for skipping unneeded chunks Data partitioning using the directory structure fastparquet bridges the gap to provide native Python read/write access with out the need to use Java. Until now, Spark's Python interface provided the only way to write Spark files from Python. Much of the time is spent in deserializing the data in the Java-Python bridge. Also, note that the times column returned is now just integers, rather than the correct datetime type. Not only does fastparquet provide native access to Parquet files[...]



Albert Hopkins: Writing autofill plugins for TeamPlayer

Tue, 06 Dec 2016 17:39:44 +0000

Background TeamPlayer is a Django-based streaming radio app with a twist. A while back it gained a feature called "shake things up" where, instead of dead silence, "DJ Ango" would play tracks from the TeamPlayer Library when no players had any queued songs. Initially this was implemented by creating a queue for DJ Ango and then filling it with random tracks. This worked but after I while I became annoyed by the "randomness" and so went about writing a few other implementations which I call "autofill strategies". These were function definitions and the autofill logic used an if/else clause to select which function to call based on what was set in the Django settings. Recently I got rid of the if/else's and instead use setuptools entry points. This also allows for third parties to write "autofill plugins" for TeamPlayer. Here's how to do it. As I said every autofill strategy is a Python function with the following signature: def my_autofill_strategy(*, queryset, entries_needed, station): This function should return a list of teamplayer.models.LibraryItem. The list should ideally have a length of entries_needed but no longer, and the returned list should contain entries from the queryset. The "should"s are emphasized because sometimes a particular strategy can't find enough entries from the queryset so it can either return a smaller list or return entries not in the queryset or both. The station argument is the teamplayer.models.Station instance for which songs are being selected. This is (almost) always Station.main_station(). Idea Regular terrestrial radio stations often play the same set of songs in rotation over and over again. This is one reason why I rarely listen to them. However I thought this would be an interesting (and easy) autofill strategy to write. Implementation Here's the idea: keep a (play)list of songs from the TeamPlayer Library for rotation, store it in a database table, and then write the autofill function to simply pick from that list. Here is the Django database model: from django.db import models from teamplayer.models import LibraryItem class Song(models.Model): song = models.OneToOneField(LibraryItem) This table's rows just point to a LibraryItem. We can use the Django admin site to maintain the list. So again the autofill function just points to entries from the list: from .models import Song def rotation_autofill(*, queryset, entries_needed, station): songs = Song.objects.order_by('?')[:entries_needed] songs = [i.song for i in songs] return songs Now all that we need is some logic to run the commercial breaks and station identification. Just kidding. Now all that is needed is to "package" our plugin. Packaging As I've said TeamPlayer now uses setuptools entry points to get autofill strategies. The entry point group name for autofill plugins is aptly called 'teamplayer.autofill_strategy'. So in our setup.py we register our function as such: # setup.py from setuptools import setup setup( name='mypackage', ... entry_points={ 'teamplayer.autofill_strategy': [ 'rotation = mypackage.autofill:rotation_autofill', ] } ) Here the entry_points argument to setup defines the entry points. For this we declare the group teamplayer.autofill_strategy and in that group we have a single entry point called rotation. rotation points to the rotation_autofill function in the module myp[...]



Obey the Testing Goat: Second Edition update: Virtualenvs, Django 1.10, REST APIs, cleaner FTs...

Tue, 06 Dec 2016 17:12:00 +0000

A brief update on my progress for the second edition. Getting there! Virtualenvs all the way down. In the first edition, I made the judgement call that telling people to use virtualenvs at the very beginning of the book would be too confusing for beginners. I've decided to revisit that decision, since virtualenvs are more and more de rigueur these days. I mean, if the djangogirls tutorial is recommending one, given that it's the most beginner-friendly tutorial on Earth, then it really must be a good idea. So there's new instructions in the pre-requisite installations chapter. Let me know if you think they could be clearer. Django 1.10 Django 1.10 doesn't introduce that many new features over 1.8, but upgrading was still pretty fiddly. Thank goodness for my extensive tests (tests for the tests in the book about testing, yes. because of course.) The main changes you'll likely to notice is in Chapter 4 where I introduce the Django Test Client, much earlier than I used to (which, through a long chain of causes, is actually because of a change to the way csrf tokens are generated). Other than that, Django 1.10 was pretty much a drop-in replacement. The main thing I'm preparing for really is the upgrade to 1.11LTS early next year. REST APIs I was thinking of having a couple of in-line chapters on building a REST API, but for now I've decided to have them as appendices. It starts with how to roll your own, including an example of how to test client-side ajax javascript with sinon, and then there's a second appendix on Django Rest Framework. These are both very much just skeleton outlines at the moment, but, still, feedback and suggestions appreciated. A cleaner flow for Chapter 6 Chapter 6 is all about rewriting an app that almost works, to be one that actually works, but trying to work incrementally all along, and using the FTs to tell us when we make progress, and warn us if we introduce regressions. I used to have just the one FT, and track progress/regressions by "what line number is the FT failing at? is it higher or lower than before?". Instead I've split out one FT that tests that the existing behaviour still works, and one FT for the new behaviour, and that's much neater I think. Next: geckodriver and Selenium 3 (uh-oh!) There are plenty more little tweaks and nice-to-have additions I can think of (React? Docker? Oh yeah, I got your trendy topics covered), but the main task that's really outstanding is upgrading to Selenium 3 and geckodriver. And the reason that's scary is because the current status of implicit waits is up for debate, and I rely on implicit waits a lot. Introducing explicit waits earlier might be a good thing (they're currently only mentioned in Chapter 20), but it would definitely add to the learning curve in the early chapters (I think they'd have to go in chapter 4 or 5, which feels very early indeed). So I'm kinda in denial about this at the moment, hoping that maybe Mozilla will reintroduce the old behaviour, or maybe I'll build some magical wrapper around selenium that just does implicit waits for you (maybe using my stale element check trick) (in my copious spare time), or maybe switch to chromedriver, or I don't know I don't want to think about it. Suggestions, words of encouragement, moral support all welcome here. In the meantime, I hope you enjoy the new stuff. Keep in touch![...]