Subscribe: l e w k . o r g
Added By: Feedage Forager Feedage Grade B rated
Language: English
bodhi  code  creator  fedora live  fedora  live  liveusb creator  liveusb  new  org  packages  python  testing  time  updates 
Rate this Feed
Rate this feedRate this feedRate this feedRate this feedRate this feed
Rate this feed 1 starRate this feed 2 starRate this feed 3 starRate this feed 4 starRate this feed 5 star

Comments (0)

Feed Details and Statistics Feed Statistics
Preview: l e w k . o r g

l e w k . o r g


Fedora 20 updates metrics, compared to F19.
Fedora 20 recently reached End Of Life. Below are metrics generated from Bodhi, along with the percent change compared to Fedora 19. 15012 updates (+8.37%) 6952 packages updated (+6.09%) 13207 stable updates (+8.59%) 250 testing updates (+8.69%) 35 pending updates (-10.26%) 1520 obsolete updates (+14.72%) 8397 bugfix updates (55.94% of updates) (+5.60%) 3605 enhancement updates (24.01% of updates) (+17.89%) 1114 security updates (7.42% of updates) (+15.80%) 1896 newpackage updates (12.63% of updates) (+6.28%) 12315 bugs resolved (+13.61%) 1603 critical path updates (10.68% of updates) (+44.41%) 1560 approved critical path updates (+43.12%) 0 unapproved critical path updates (+0%) 144 critical path security updates (16.78% of security updates, 14.02% of critpath updates) (-15.79%) 5455 updates received feedback (36.34% of updates) (+34.29%) 1259 +0 comments (+41.62%) 11579 +1 comments (+47.79%) 678 -1 comments (+29.14%) 1195 unique authenticated karma submitters (+15.35%) 49 proventesters (+2.08%) 1681 +1's from proventesters (+5.13%) 96 -1's from proventesters (+41.18%) 5 critpath updates with conflicting proventesters (0.31% of critpath updates) (+25%) dnf-0.4.18-1.fc20 kernel-3.13.4-200.fc20 perl-5.18.4-290.fc20 sudo-1.8.11p2-1.fc20 testdisk-6.14-3.fc20 15 critpath updates with positive karma and negative proventester feedback (0.94% of critpath updates) (+200%) createrepo-0.9.9-23.fc20 dnf-0.4.15-1.fc20 dnf-0.4.18-1.fc20 dracut-037-10.git20140402.fc20 gnome-shell-3.10.4-6.fc20 kernel-3.11.10-301.fc20 kernel-3.13.3-201.fc20 kernel-3.13.4-200.fc20 kernel-3.14.2-200.fc20 libsolv-0.6.0-0.git05baf54.fc20 llvm-3.4-10.fc20 PackageKit-0.8.14-2.fc20 perl-5.18.4-290.fc20 sudo-1.8.11p2-1.fc20 testdisk-6.14-3.fc20 630 critpath updates with positive karma and positive proventester feedback (39.30% of critpath) (+36.07%) 304 anonymous users gave feedback (2.26%) (+49.75%) 2396 updates reached the stable karma threshold (18.14%) (+36.60%) 11200 updates reached the minimum time in testing threshold (84.80%) (+1.91%) 8259 went from testing to stable *without* karma (63.79%) (-3.79%) 33 updates were pushed to stable with negative karma (0.25%) (-10.81%) 57 critical path updates pushed to stable *without* karma (-60.96%) Time spent in testing: mean = 13 days (+8.33%) median = 9 days (+12.5%) mode = 8 days (+0%) [...]

Release Tools and Infrastructure FAD
The Release Tools and Infrastructure Fedora Activity Day happened recently at the Red Hat office in Westford, Massachusetts. The goal was to bring our release tooling and processes up to speed with the current and future demands of the Fedora Project. Since there are a ton of moving parts of the Fedora Release Engineering community that need work, many of us split out into groups to tackle various components. Even though there were a ton of shiny tasks that I would have loved to dive into, like pungi4, createrepo_c, koji 2.0, etc, I felt like my cycles were best spent working on Bodhi 2.0. Bodhi is a critical piece of infrastructure that has been in production for over 7 years now, and it's initial incarnation was deployed internally to Red Hat for Fedora Core prior to that. It is currently a cross-section between releng, qa, and infra, and is responsible for shipping updated bits to users while facilitating community feedback. It started with pushing out updated RPMs initially, but since F21 it has also started updating Atomic OSTrees as well. With the number of requirements for updated deliverables growing each release, it's clear that our current architecture does not scale with this appropriately. The next generation of Bodhi has been in the works for a while now, and is almost ready for testing in staging. So I started off the FAD by getting the current bodhi2 stack built for RHEL7. This turned out to be a non-trivial task, as I had to build a bunch of complex dependencies like python-cryptography, python-six, and python-py/pytest (which have circular deps on eachother). Thankfully, I now have a bodhi2 copr setup for building the extra pieces of the stack until they are available in RHEL7+EPEL. I then began writing a new bodhi2 ansible role and got an instance deployed to staging. There is still a bunch of work to be done before the staging instance is usable, but that mostly revolves around configuration and hopefully shouldn't be too far off. Once we have something we can start playing with, there is going to be a bunch of work needed to port the existing tools over to use the new API. I'm hoping we can keep the python-fedora bindings "stable", but there are still plenty of other things that hit the JSON API directly that will most likely need tweaking. As far as future goals for the Bodhi architecture goes, I'd really like to see us split it up into a couple of different components. Ideally, I think the majority of it would revolve around the web frontend & RESTful JSON API that facilitates community testing. There's already a pretty awesome ecosystem that has been built around this, with client-side QA tools like fedora-easy-karma and fedora-gooey-karma which make it easy people test and give feedback to things in updates-testing that they have installed. Then there is "The Masher" backend, which currently does most of the heavy lifting. Most of this could ideally be pushed into pungi4 and koji plugins, for things like mashing updates repos, composing ostrees, generating updated images/installers, etc. This way, bodhi would simply handle tagging things in koji and then orchestrate the creation of the updated bits with pungi4, update/close bugs, generate updateinfo metadata, etc. Then we can finally leverage our build farm to perform the long-running composition tasks in parallel, instead of doing them all linearly on a single releng machine. So bodhi would still need to handle the orchestration of these processes, but I could also see us pushing this stuff into the theoretical "ComposeDB" idea that we have been tossing around as well. This could then potentially take care of rebuilding/composing whatever needs to be updated based on what has changed. I wasn't directly involved with the detailed requirements discussions between Dennis, Ralph, etc.. but from my perspective I am really excited about the idea of a central tool that knows about all release artifacts, what is contained within them, and how to produce them. Then when something like a critical security u[...]

Atomic OSTree updates for Fedora 21
With the new Fedora 21 Cloud Atomic Host released, we needed a way to continuously update the OSTree during the Fedora updates process. The rawhide & branched nightly tree composes are triggered by cron jobs, but we needed to somehow tie it into the Bodhi push process for F21. There are a number of ways to compose OSTrees, but for this task we needed a lot more control and flexibility than things like the rpm-ostree-toolbox give us. So I wrote a new tool called (for the lack of a better name) the fedmsg-atomic-composer. It is comprised of a command-line tool, a Python API, as well as a fedmsg consumer that can trigger tree composes whenever Fedora repositories are updated. Here's how it works: The default contains all of the release information, paths and commands, which can easily be modified to customize your tree. It starts by rsyncing the canonical production OSTree repository locally, if one exists. The corresponding fedora-atomic.git branch is cloned, which contains the parent fedora-atomic-docker-host.json treefile that we inherit from. We then dynamically generate the mock configuration and yum repo files from Mako templates based on the values. The mock chroot is then initialized with the latest OSTree stack from our repositories. With the latest version of mock, it will use a systemd-nspawn container instead of a chroot. The OSTree is then composed in the mock container with our custom treefile Upon completion, the OSTree summary file is updated. This is a serialized GVariant that describes the available branches along with other metadata. During the recent MirrorManager 2 FAD, Ralph implemented native support for OSTrees, which relies on this file. The updated tree is then synced back out to production, where users will then be able to atomic upgrade and pull in the changes. From here, hooking this tool into the Bodhi push process was fairly simple. All we needed to do was to update the repo URLs in our to point to our freshly mashed local repositories, and then we're good to go. So far the tree composes has been churning along smoothly, and we have yet to encounter any major issues. I think the next step for our Atomic infrastructure is probably to get the new MirrorManager2 deployed so we can start to balance the load a bit. [...]

liveusb-creator 3.13 release!

A new version of the liveusb-creator has just hit the Fedora repos! It's been a long time since there has been any major changes to the tool, but I managed to find a couple of days last week to implement some much-needed functionality.


This version features a new destructive installation method, which uses dd to directly copy the the entire ISO to to your device, overwriting your existing data and partitions. This method tends to be much more reliable, since we don't have to crack open the ISOs, mess with partition flags, or configure and install the bootloader ourselves. This also means that DVD, netinst, and boot images finally work with this new method as well. You can still perform the traditional non-destructive install that the liveusb-creator has always done, and it remains the default.

Unfortunately there is not a Windows build for this release just yet, as I hit some issues getting dd.exe to work properly in my virtual machine. Hopefully I'll be able to figure it out at some point next week. If any Windows folks are out there and want to help out, checkout the Developers Guide for instructions on setting things up.

There are also a number of other features and fixes in this release, checkout the wiki for more details. The source code can also be found on GitHub.

Bodhi2 Fedora Activity Day
The Bodhi2/Taskotron Fedora Activity Day happened earlier this month! A bunch of us gathered in Denver for a few days and worked on some of our critical releng & qa infrastructure. The hackfest was held in a conference room in my apartment building, which worked out quite nicely for the amount of people that we had. The hotel was right up the road, and we were able to walk to a lot of awesome spots, like the 1UP Barcade :). It was great to have folks from various corners of Fedora together in the same room for a few days. As it is, we get a lot done remotely, but being able to leverage the high-bandwidth face-to-face time is extremely valuable, especially when coming to consensus on difficult architectural decisions. We used Gobby to collaborate on a long list of action items, and chipped away most of it. Thankfully, Bodhi has enough layers that we were all able to split up and dive into different corners of the code without stepping on each others' toes. Up until now, our releng stack in staging has always been less than useful. We've never able to do a full build->update->push->run, and have had to rely on testing certain codepaths in production. Not only that, but Bodhi's "masher" never really had proper unit tests, so pushing out major changes to that stack has always been quite unpleasant. Thankfully, Kevin and Dennis worked on our staging setup and made it so we can actually use it. I made a lot of headway on porting the Bodhi masher to a new fedmsg-driven architecture, while writing unit tests for it along the way. I'm hopeful that we can write tests for every part of the "push" process, and then optimize the hell out of it :) While Mathieu and I mainly focused on back-end hacking, Ralph and Ricky made some fantastic headway on the front-end. Ralph started working on the revamped New Update Form, which is already significantly better than the original. The idea here is that the maintainer should only need to provide the package name(s), and Bodhi will automatically find the corresponding candidate builds, related bugs, and eventually it will pull in all candidate deps as well (or tell if you if any need to be rebuilt). It would also be very convenient to be able to "push this entire stack again". Ideally, I'd love to make it so that folks maintaining large stacks, like GNOME, shouldn't need to use a Google doc to coordinate their mega-updates ;) Ralph also started revamping the new karma system (check out his screencast here). We don't have any of the policy in place to enforce it yet, but eventually we'd like the maintainers to be able to define custom policy constraints for their updates. For example, you could say "only allow this update to go to the stable repo once this specific bug or test case has been confirmed as fixed/passing". Ricky made lots of improvements to the Release profiles and Updates/Comments/User pages, which are all looking great. He also created a Bodhi news feed on the front page using the fedmsg datagrepper widget that Ralph blogged about recently. Other front-end improvements include libravatar support for all users, proper markdown rendering with previews and image support, and of course a konami code easter-egg ;) I was going to post a bunch of screenshots here, but Ralph just deployed a development instance of Bodhi2 that folks can play around with instead: (it's a dev instance, so don't expect it to always be up/functional). Some other corners of Bodhi that have seen improvements recently: The API. The Bodhi webapp is written in Python using the excellent Pyramid web framework. On top of that we are using Cornice, which makes it really easy to build & document standards-compliant web services with Pyramid. Thanks to colander validation/deserialization schemas and our custom ACLs and validators, we are able to write dead-simple views that can safely assume that all of the data we are dealing with is valid[...]

Keeping your finger on the pulse of the Fedora community
For those who haven't been keeping up with all of the awesome code Ralph Bean has been churning out lately, be sure to checkout Hop on #fedora-fedmsg on Freenode or load up busmon to see it in action. Not all of the Fedora Infrastructure services currently fire off fedmsgs, but we're getting very close. This technology is built on top of Moksha, which I created many years ago while writing the first version of the fedoracommunity app. It's come a long way since then, and now can speak ØMQ over WebSockets, as well as AMQP and STOMP over Orbited. Now the time has finally come to bring Moksha to the desktop! Introducing fedmsg-notify fedmsg-notify lets you get realtime desktop notifications of activity within the Fedora community. It allows you to tap into the firehose of contributions as they happen and filter them to your liking. It works with any window manager that supports the notification-spec, however I've only seen the gravatars show up using GNOME. For GNOME Shell users, you can [optionally] install gnome-shell-extension-fedmsg, and then enable it with the gnome-tweak-tool or by running `gnome-shell-extension-tool -e` (and then hit alt+f2 and type 'r' to reload the shell). You will then be graced with the presence of The Bus: For those who aren't running GNOME shell, you can simply yum install fedmsg-notify, and then run fedmsg-notify-config, or launch it from your Settings menu. Due to a dependency on Twisted's gtk3reactor, fedmsg-notify is currently only available on Fedora 18 and newer. The first tab shows you all services that are currently hooked into fedmsg. As we add new ones, the gui will automatically display them. These services are defined in the fedmsg_meta_fedora_infrastructure package. The Advanced tab lets you further customize what messages you want to see. The "Bugs that you have encountered" option will display all messages that reference any Bugzilla numbers for crashes that you have hit locally with ABRT. The other filters involve querying your local yum database or the PackageDB. Under the hood The fedmsg-notify-daemon itself is fairly minimal (see At it's core, it's just a Twisted reactor that consumes ØMQ messages. Moksha does all of the heavy lifting behind the scenes, so all we really have to do is specify a topic to subscribe to and define a consume method that gets called with each message. This is essentially just a basic Moksha Consumer with some fedmsg + DBus glue. class FedmsgNotifyService(dbus.service.Object, fedmsg.consumers.FedmsgConsumer): topic = 'org.fedoraproject.*' def consume(self, msg): … The daemon will automatically startup upon login, or will get activated by DBus when enabled via the GUI. When a message arrives, it filters it accordingly, downloads & caches the icons, [optionally] relays the message over DBus, and then displays the notification on your desktop. The API for writing custom filters is dead simple (see Here is an example of one: class MyPackageFilter(Filter): """ Matches messages regarding packages that a given user has ACLs on """ __description__ = 'Packages that these users maintain' __user_entry__ = 'Usernames' def __init__(self, settings): self.usernames = settings.replace(',', ' ').split() self.packages = set() reactor.callInThread(self._query_pkgdb) def _query_pkgdb(self): for username in self.usernames:"Querying the PackageDB for %s's packages" % username) for pkg in PackageDB().user_packages(username)['pkgs']: self.packages.add(pkg['name']) def match(self, msg, processor): packages = processor.packages(msg) for package in self.packages: if package in packages: [...]

Wielding the ANU Quantum Random Number Generator
Last week Science Daily published an article that caught my attention titled 'Sounds of Silence' Proving a Hit: World's Fastest Random Number Generator. The tl;dr is that researchers at the ANU ARC Centre of Excellence for Quantum Computation and Communication Technology created a blazing fast random number generator based on quantum fluctuations in a vacuum. Thankfully, these awesome scientists are giving their data away for free, and they even provide a JSON API. In an effort to make it simple to leverage this data, I created a new project: quantumrandom. It provides a qrandom command-line tool, a Python API, and also a /dev/qrandom Linux character device. Installing $ virtualenv env $ source env/bin/activate $ pip install quantumrandom Using the command-line tool $ qrandom --int --min 5 --max 15 7 $ qrandom --binary ���I�%��e(�1��c��Ee�4�������j�Կ��=�^H�c�u oq��G��Z�^���fK�0_��h��s�b��AE=�rR~���(�^ +a�a̙�IB�,S�!ꀔd�2H~�X�Z����R��.f ... $ qrandom --hex 1dc59fde43b5045120453186d45653dd455bd8e6fc7d8c591f0018fa9261ab2835eb210e8 e267cf35a54c02ce2a93b3ec448c4c7aa84fdedb61c7b0d87c9e7acf8e9fdadc8d68bcaa5a ... Creating /dev/qrandom quantumrandom comes equipped with a multi-threaded character device in userspace. When read from, this device fires up a bunch of threads to fetch data. Not only can you utilize this as a rng, but you can also feed this data back into your system's entropy pool. In order to build it's dependencies, you'll need the following packages installed: svn gcc-c++ fuse-devel gccxml libattr-devel. On Fedora 17 and newer, you'll also need the kernel-modules-extra package installed for the cuse module. pip install ctypeslib hg+ sudo modprobe cuse sudo chmod 666 /dev/cuse qrandom-dev -v sudo chmod 666 /dev/qrandom By default it will use 3 threads, which can be changed by passing '-t #' into the qrandom-dev. Testing the randomness for FIPS 140-2 compliance $ cat /dev/qrandom | rngtest --blockcount=1000 rngtest: bits received from input: 20000032 rngtest: FIPS 140-2 successes: 1000 rngtest: FIPS 140-2 failures: 0 rngtest: FIPS 140-2(2001-10-10) Monobit: 0 rngtest: FIPS 140-2(2001-10-10) Poker: 0 rngtest: FIPS 140-2(2001-10-10) Runs: 0 rngtest: FIPS 140-2(2001-10-10) Long run: 0 rngtest: FIPS 140-2(2001-10-10) Continuous run: 0 rngtest: input channel speed: (min=17.696; avg=386.711; max=4882812.500)Kibits/s rngtest: FIPS tests speed: (min=10.949; avg=94.538; max=161.640)Mibits/s rngtest: Program run time: 50708319 microseconds Adding entropy to the Linux random number generator sudo rngd --rng-device=/dev/qrandom --random-device=/dev/random --timeout=5 --foreground Monitoring your available entropy levels watch -n 1 cat /proc/sys/kernel/random/entropy_avail Python API The quantumrandom Python module contains a low-level get_data function, which is modelled after the ANU Quantum Random Number Generator's JSON API. It returns variable-length lists of either uint16 or hex16 data. >>> quantumrandom.get_data(data_type='uint16', array_length=5) [42796, 32457, 9242, 11316, 21078] >>> quantumrandom.get_data(data_type='hex16', array_length=5, block_size=2) ['f1d5', '0eb3', '1119', '7cfd', '64ce'] Based on this get_data function, quantumrandom also provides a bunch of higher-level helper functions that make easy to perform a variety of tasks. >>> quantumrandom.randint(0, 20) 5 >>> quantumrandom.hex()[:10] '8272613343' >>> quantumrandom.binary()[0] '\xa5' >>> len(quantumrandom.binary()) 10000 >>> quantumrandom.uint16() numpy.array([24094, 13944, 22109, 22908, 34878, 33797, 47221, 21485, 37930, ...], dtype=numpy.uint16) >>> quantumrandom.uint16().data[:10] '\x87\x7fY.\xcc\xab\xea\r\x1c`' Follow quantumrandom on GitHub:[...]

Red Hat OpenShift Express & The Leafy Miracle
Red Hat made a lot of awesome announcements this week at The Red Hat Summit, one of which being OpenShift. I had the opportunity to play with the internal beta for a little while now, and I must say that as a developer I am extremely impressed with the service. Just being able to git push my code into to the cloud drastically simplifies large-scale software deployment, and makes it so I don't even have to leave my development environment. I figured out a way to get TurboGears2 and Pyramid running on OpenShift Express, and documented it here and here. After that, I proceeded to write my very first Pyramid application. [ The Leafy Miracle ] In memory of the proposed [and rejected] Fedora 16 codename "Beefy Miracle", this little app is called "Leafy Miracle". [ Features & Tech ] Written in Python using the Pyramid web framework SQLAlchemy database model of Yum Categories, Groups, Packages, Dependencies, and Dependants Interactive graph widget, using ToscaWidgets2 and the JavaScript InfoVis Toolkit Package mouse-over menus linking to downloads (community), code (gitweb), bugs (bugzilla), acls (pkgdb), builds (koji) and updates (bodhi). Deep linking, which allows you to use the browser back/forward buttons while navigating the tree, as well as share or bookmark links to specific nodes Search bar with auto-completion of packages, categories, and groups jQuery under the hood, powering the widgets [ Running ] sudo yum -y install python-virtualenv git clone git:// && cd leafymiracle virtualenv env && source env/bin/activate python develop python leafymiracle/ paster serve development.ini [ Code ] git clone git:// [ Props ] Mad props go out to RJ Bean, who helped me write this app. He is responsible for writing a ton of amazing Python widgets for various JavaScript visualization libraries. You can see some demos of them here: [...]

git clone all of your Fedora packages

After doing a fresh Fedora 15 install on my laptop last night, I wanted to quickly clone all of the packages that I maintain. Here is a single command that does the job:

python -c "import pyfedpkg; from fedora.client.pkgdb import PackageDB; [pyfedpkg.clone(pkg['name'], '$USER') for pkg in PackageDB().user_packages('$USER')['pkgs']]"

Fedora Photobooth @ SXSW
This is the first year that Fedora will have a booth at SXSW! Sadly, I am not going to be attending since it conflicts with PyCon. However, my code will be running at our booth. Usually the Fedora booth at conferences is comprised of a bunch of flyers, media, swag, and some people to help answer questions and tell the Fedora story. However at SXSW, things are going to be a little different. Aside from the amazing flyers that Máirín created, there will also be a Fedora Photobooth. Someone (probably Spot or Jared) will be dressed in a full Tux costume, and people can come and get their photo taken with them. Spot came to me the other day and asked if I could write some code to streamline the whole process. An hour or so later, was born. There are definitely lots of improvements that can be made, but here is what it currently does in its initial incarnation: Uses gphoto2 to automatically detect your camera When Enter is pressed, it snaps a photo and downloads it locally A Fedora watermark is applied to the bottom right corner of the image The photo is uploaded to a server A QRCode is generated that points to the image URL A TinyURL is generated for the image HTML is generated that shows the image, the QRCode, and TinyURL The page is then displayed in the web browser In Action See Mo's blog for photos of this code in action at the Fedora SXSW booth! * SXSW Expo Day 1 from the show floor * SXSW Expo Day 2 * A Beefy, Miraculous Day at SXSW (Expo Day 3) The Code I threw this in a git repo and tossed it up on GitHub: #!/usr/bin/python # - version 0.3 # Requires: python-imaging, qrencode, gphoto2, surl # Author: Luke Macken # License: GPLv3 import os import surl import Image import subprocess from uuid import uuid4 from os.path import join, basename, expanduser # Where to spit out our qrcode, watermarked image, and local html out = expanduser('~/Desktop/sxsw') # The watermark to apply to all images watermark_img = expanduser('~/Desktop/fedora.png') # This assumes ssh-agent is running so we can do password-less scp ssh_image_repo = '' # The public HTTP repository for uploaded images http_image_repo = '' # Size of the qrcode pixels qrcode_size = 10 # Whether or not to delete the photo after uploading it to the remote server delete_after_upload = True # The camera configuration # Use gphoto2 --list-config and --get-config for more information gphoto_config = { '/main/imgsettings/imagesize': 3, # small '/main/imgsettings/imagequality': 0, # normal '/main/capturesettings/zoom': 70, # zoom factor } # The URL shortener to use shortener = '' class PhotoBooth(object): def initialize(self): """ Detect the camera and set the various settings """ cfg = ['--set-config=%s=%s' % (k, v) for k, v in gphoto_config.items()]'gphoto2 --auto-detect ' + ' '.join(cfg), shell=True) def capture_photo(self): """ Capture a photo and download it from the camera """ filename = join(out, '%s.jpg' % str(uuid4())) cfg = ['--set-config=%s=%s' % (k, v) for k, v in gphoto_config.items()]'gphoto2 ' + '--capture-image-and-download ' + '--filename="%s" ' % filename, shell=True) return filename def process_image(self, filename): print "Processing %s..." % filename print "Applying watermark..." image = self.watermark(filename) print "Uploading to remote server.[...]

FUDCon 2011 Tempe
I meant to get this summary out the door weeks ago, however, I didn't want to distract from my post-FUDCon productivity :) Another FUDCon has come and gone. This time it was in gorgeous Tempe, Arizona. It was my first time in AZ, and I must say that I was thoroughly impressed. It was truly a great location for FUDCon. Thanks to the epic snowstorm of doom, I also was stuck there for a few extra days Live Widgetry Before the conference I decided to throw together a little live widget that scrolls all of the new posts tagged with #FUDCon. Thanks to the mad design skillz of mizmo, and the feed aggregation and real-time web sockets of Moksha, I was able to throw it together pretty quickly. I plan on taking this code and integrating it in the existing fedoracommunity dashboard and hooking up many different fedora-related feeds to it. Sessions AutoQA Day 1 of the FUDCon sessions were quite interesting. I got a chance to learn a bit more about the exciting AutoQA project, which is coming along nicely. You'll be seeing AutoQA commenting on bodhi updates soon enough. Security Lab I caught Joerg Simon's session on the Fedora Security Lab. It was exciting to see how the Security Spin has evolved ever since I created it back in 2007 for a project in my forensics class. It was also interesting to learn more about OSSTMM. Spins The next session I attended was about the future of spins. Almost everyone agreed that Spins are useful and a valuable part of Fedora. The problems seem to mostly lie in governance/policy and a lack of communication and coordination between the Spins SIG and QA/Releng/Infrastructure. Once of my ideas below may help with this a little bit. The Next Big Project Day 2 was comprised of more sessions, including my team's "Next Big Project" proposals. Of course, the Fedora RPG that Spot and Mo talked about was definitely a hot topic, and got a lot of people excited. I talked about a handful of project ideas that I would like to work on in the future (actually, I have code written for most of them already). Here is a quick rundown: Real-time Infrastructure I want to see us deploy an AMQP message broker inside our production infrastructure. Then, we hook up all of our existing services and have them fire off messages when various events occur (koji builds, bodhi updates, pkgdb additions/removals/changes, git hooks, planet feeds, wiki edits, etc). From here, using some realtime web technology that we created, we could easily expose these message queues via a live dashboard that lets you filter and navigate the stream of activity, along with providing real-time metrics. We can also create desktop notification widgets, so you can get popup bubbles for things that you care about. This is also key to the whole RPG as well. In order to build a game based on Fedora workflows, we need an underlying expert system that knows what actions can be taken within fedora, how they are accomplished, and who is getting them done. Meeting app Currently after a meeting, our Meetbot spits out the logs and an overview in txt/html/rst to a directory on Trying to track down who agreed to what when, or even to see what a given team has been up to over the past couple of months, is very tedious. The data is there, now we just need to make it useful. I would love to see a frontend for this sytem that tracked meetings by team/people/topics/projects, kept track of actions and held people accountable for what they say they are going to do (and make it easy for people to say "I need help with this", or "I don't have time to finish this"), making it simple to go from an #idea in a meeting to implementation. There is so much great data in these logs, and I think we can do some awesome thin[...]

liveusb-creator 3.9.3 windows release

I spent the majority of yesterday at a DOS prompt. Thankfully, it wasn't as painful as it sounds, as git, vim and Python make Windows development quite tolerable.

Anyway, I was finally able to track down and fix a couple of major bugs in the liveusb-creator on Windows XP and 7, and I pushed out a new build yesterday with the following changes:

  • Rebuilt with Python 2.7 and the latest PyQt4/pywin32/py2exe/NSIS
  • Update to syslinux 4.03 in our Windows package, which works with Fedora 14
  • Determine if we are running with admin privs, and warn otherwise
  • Fix how and where we put our error logs
  • Update our list of Fedora & Sugar releases
  • Download releases to Downloads or My Documents
  • Various Windows path-related fixes
  • Translation updates

Windows users, download it here:

Professors' Open Source Summer Experience @ RIT
Last week Red Hat put on a Professors' Open Source Summer Experience (POSSE) at RIT. Being an alumni, I was excited by the opportunity to be able to go back up to The ROC and teach some of the people that taught me. Going into it, I really had no idea what to expect. All I knew is that I was going to help lead the 'deep dive' section of the course, where I would teach professors how to dive in head first and get productively lost in a strange codebase. This is not something that can be accomplished with a set of powerpoint slides. Teaching how to hack on open source requires that you emerse yourself into a codebase, and bring your students with you. The previous POSSE at Worcester State dove into the Sugar Measure Activity, and we were going to do the same. Measure is an activity that turns the computer into an oscilloscope. Signals from the microphone (and sensors) can be plotted in time and frequency domains. I had never used this activity, let alone hacked on it before. I've also never done any sugar activity development, aside from some tweaking of the OpenVideoChat, so I really had no idea what I was getting myself into. The obvious first step was to get it running. All of us were able to start the activity in virtual machines or emulators, except for one install which hit some odd errors upon startup. We were able to quickly track the bug down to a stray return statement in __init__ before some critical initialization code. Right after we fixed the problem we noticed that Walter Bender had already fixed this issue a few hours earlier. After a git pull, we were up and running. Once we all got the activity running, we took a look at the bug list to see if there was any low-hanging fruit for us to tackle. Since the previous POSSE had already done some work on this activity the week before, there were not any trivial tickets left in the queue. So, in that case, we dove head first into the hardest one, "Measure activity gets stuck after recording". This ticket had very little information, and no log output, so we were on our own to try and track it down. We were able to reliably reproduce the issue on the XO-1.5, but not on the 1.0. In our virtual machines we hit it sporadically. We all agreed that it felt like a race-condition, most likely due to threading. So we started instrumenting the code and adding some debugging statements to try and figure out which line of code was the culprit. In our efforts to scatter print statements all over the place to try and determine the code path, we noticed that none of our output was hitting the logs. When you have no idea if your code is even being run or not, don't be ashamed to throw in a raise Exception("WTF?!"). We finally realized that since the activity would freeze, it was never able to flush stderr/stdout to the log. A quick find/replace regex later, and we were using the proper logging module and seeing our debugging output hit the logs. We were then able to track the bug down to a line of code that uses GTK to try and get the coordinates of the parent window. At this point, since none of us were GTK experts, we had to go upstream. So, I dropped into #fedora-devel and asked. Within 10 minutes I had responses from 3 different GTK hackers. One was a typical RTFM response, which humored the professors (who are very used to hearing/saying this), but the others pointed us in the right direction. One mentioned that gtk.gdk calls probably should not be done in a seperate thread. So, a suggested workaround was to add gtk.threads_enter()/gtk.threads_leave() calls before running any gtk.gdk code in the thread. A 2-line patch later, and we had squashed the bug. We eventually bumped into the inevitable typo in some[...]

Fedora Updates Report
I recently wrote some code to generate detailed statistics of Fedora & EPEL updates within bodhi. Eventually this will be auto-generated and exposed within bodhi itself, but for now here are the initial metrics. This report definitely conveys the shortcomings in how we currently utilize bodhi for "testing" updates, however, it does show us improving with each release. For Fedora 13, we implemented the No Frozen Rawhide process with improved Critical Path policies, which were definitely a success. With these enhanced procedures, along with the upcoming implementation of AutoQA and the new Package update acceptance criteria, I think we'll see these numbers drastically improve in the future. You can find the code that generates these statistics here:, If you have any ideas or suggestions for different types of metrics to generate, or if you find any bugs in my code, please let me know. Bodhi Statistics Report (Generated on June 8th, 2010) ===================================================== Out of 17412 total updates, 2958 received feedback (16.99%) Out of 1045 total unique karma submitters, the top 30 are: * notting (424) * mclasen (366) * jkeating (321) * adamwill (283) * cwickert (161) * rdieter (159) * pbrobinson (141) * kevin (141) * cweyl (122) * tomspur (119) * mtasaka (110) * xake (97) * cschwangler (86) * kwright (84) * peter (83) * hadess (80) * michich (72) * tagoh (69) * pfrields (69) * bpepple (69) * iarnell (68) * lkundrak (66) * shinobi (65) * sundaram (64) * spot (62) * pravins (62) * markmc (62) * thomasj (61) * smooge (60) * fab (59) ================================================================================ Fedora 13 ================================================================================ * 3562 updates * 3065 stable updates * 427 testing updates * 62 pending updates * 8 obsolete updates * 2371 bugfix updates (66.56%) * 745 enhancement updates (20.92%) * 89 security updates (2.50%) * 357 newpackage updates (10.02%) * 410 critical path updates (11.51%) * 333 critical path updates approved * 1155 updates received feedback (32.43%) * 12120 +0 comments * 2477 +1 comments * 155 -1 comments * 595 unique authenticated karma submitters * 133 anonymous users gave feedback (1.57%) * 2261 out of 3562 updates went through testing (63.48%) * 1317 testing updates were pushed *without* karma (58.25%) * 21 critical path updates pushed *without* karma * Time spent in testing: * mean = 11 days * median = 9 days * mode = 7 days * 4 updates automatically unpushed due to karma (0.11%) * 0 of which were critical path updates * 231 updates automatically pushed due to karma (6.49%) * 2 of which were critical path updates * Time spent in testing of updates that were pushed by karma: * mean = 11 days * median = 7 days * mode = 7 days * Time spent in testing of updates that were unpushed by karma: * mean = 9 days * median = 5 days * mode = 5 days * 2445 packages updated (top 10 shown) * selinux-policy: 13 * jd: 12 * 12 * gdb: 12 * ibus-pinyin: 11 * nautilus: 10 * kernel: 10 * evolution: 9 * libfm: 9 * libmx: 9 ================================================================================ Fedora 12 ================================================================================ * 4844 updates * 4291 stable updates * 371 testing updates * 113 pending updates * 69 obsolete updates * 2905 bugfix updates (59.97%) * 1054 enhancement updates (21.76%) * 201 security updates (4.15%) * 684 newpackage updates (14.12%) * 407 critical path updates (8.40%) * 960[...]

liveusb-creator trojan in the wild

(image) I've been noticing many different copies of my Windows liveusb-creator popping up on various sketchy-looking download sites. The majority of these copies contain a variant of the Vundo Trojan.

"Vundo, or the Vundo Trojan (also known as Virtumonde or Virtumondo and sometimes referred to as MS Juan) is a Trojan horse that is known to cause popups and advertising for rogue antispyware programs, and sporadically other misbehavior including performance degradation and denial of service with some websites including Google and Facebook."

So, if you downloaded a copy of the Windows liveusb-creator from anywhere other than -- you could be infected. Apparently the latest variation of this trojan is undetectable by most antivirus (although, clamav was able to recognize the one that I found), so you may need to look around for some of the common symptoms. There is apparently a tool that will remove this trojan which can be found here, however I have not tested it and cannot vouch for its validity.

If anyone was actually hit by this, I'd be interested to hear about it.

Also, to state the blatantly obvious: only download the liveusb-creator from the homepage!

Fedora Community Statistics

(image) I'm pleased to announce that version 0.4.0 of the Fedora Community dashboard has just hit production. Along with the usual batch of bugfixes, this release contains a new 'Statistics' section that contains metrics from a variety of different pieces of Fedora Infrastructure.

Thanks goes to Ian Weller for the wiki stats, Seth Vidal for the torrent stats, and Matt Domsch & Jef Spaleta for the map generation code. I ended up writing the updates metrics, package stats, and users/mirrors widgets. Enjoy!


nose 0.11

I know nose 0.11 is old news, but I've only recently discovered it's new multiprocess module.

lmacken@tomservo ~/bodhi $ nosetests
Ran 96 tests in 725.111s


lmacken@tomservo ~/bodhi $ nosetests --processes=50
Ran 96 tests in 10.915s


Nose 0.11 is already in rawhide, and will soon be in updates-testing.

Note to self (and others): Buy the nose developers beer at PyCon next month

My dream machine
In an effort to optimize my home office, I recently donated my server rack to a local Boston record label, to hold their Red Hat servers. I'm also in the process of donating all of my computer hardware for re-use/recycling (~10 or so frankenstein boxen). So, once I clear everything out, I'm going to replace it with a new machine. I usually sit in front of 1-3 laptops (thindpads and XOs) on the daily, and I absolutley love them, but I need something beefier. I do most of my work on remote machines, but I have found that I still spend too much time waiting on computers. I'm not much of a gamer, so I probably don't need too high-end of a graphics chip, let alone SLI/Crossfire. The extent of my gaming these days consists of the occassional wesnoth, open arena, nethack, and my current favorite Cube 2: Sauerbraten. I just want a card that will work well in Linux, ideally without having to install proprietary drivers. Anyway, I haven't built a desktop machine from the ground up in 12 years, and I've been out of the hardware game for a long time, so let me know what's good! Here is what I've been looking at so far... Intel DX58SO Extreme Series X58 ATX Tri-Channel DDR3 16GB SLI or CrossFireX LGA1366 Overclocking Utility Desktop Board or ASUS Rampage II Extreme LGA1366 Intel X58 DDR3-1600 ATX Motherboard Intel Core i7 975 Extreme Edition 3.33GHz 8M L3 Cache LGA1366 Desktop Processor Cooler Master V8 Nickel Plated Copper Base Aluminum Fins 8 Heatpipes Core i7 1366 CPU Cooler - (RR-UV8-XBU1-GP) Corsair 6 GB Dominator GT PC3-16000 2000Mhz 240-pin Triple Channel DDR3 Memory Kit Corsair CMPSU-1000HX 1000-Watt HX Professional Series 80 Plus Certified Power Supply Intel X25-E Extreme SATA Solid-State Drive or Corsair 256 GB Internal Solid State Drive (SSD) CMFSSD-256GBG2D DELL ULTRASHARP 3008WFP - 30-inch Widescreen Flat Panel Monitor EVGA 01G-P3-1180-AR GeForce GTX285 1024 MB DDR3 PCI-Express 2.0 Graphics Card Endurapro Cooler Master HAF 932 High Air Flow ATX Full Tower Case Also, if you appreciate my software and want me to write it faster... donations are accepted ;) [...]

RIP Fedora 10
Fedora 10 (Cambridge) (2008-11-25 -- 2009-12-17) Features Updates Source: bodhi Fedora 10 Updates Most updates per developer in Fedora 10 Most Updated Packages in Fedora 10 Packages with best karma Top Fedora 10 testers Most tested Fedora 10 packages Torrent Source: fedoracommunity (upcoming release) Torrent NameNumber of completed downloads Fedora-10-i386-DVD 112,807 Fedora-10-x86_64-DVD 65,965 Fedora-10-i386-CDs 10,621 Fedora-10-ppc-DVD 6,851 Fedora-10-source-DVD 3,740 Fedora-10-x86_64-CDs 3,141 Fedora-10-ppc-CDs 1,336 Fedora-10-i686-AOS 666 Fedora-10-source-CDs 662 Fedora-10-i686-Live 599 Fedora-10-x86_64-Live 336 Fedora-10-x86_64-AOS 274 Fedora-10-i686-Live-KDE 201 Fedora-10-x86_64-Live-KDE 78 Fedora-10-i686-Live-XFCE 37 Fedora-10-i686-Live-Developer 13 Fedora-10-i686-Live-FEL 12 Fedora-10-x86_64-Live-XFCE 5 Fedora-10-i686-Live-broffice 3 Fedora-10-x86_64-Live-Developer 3 Fedora-10-x86_64-Live-FEL 2 Fedora-10-x86_64-Live-edu-math 1 Fedora-10-i686-Live-edu-math 1 Fedora-10-x86_64-Live-broffice 0 Total 207,354 Yum Data Source: wiki/Legacy_statistics Connections to yum Week Dates New Unique IPsTotal Unique IPsTotal compared to F9 1 2008-11-25 -- 2008-12-0167,421 67,421 73% 2 2008-12-02 -- 2008-12-0881,674 149,095 97% 3 2008-12-09 -- 2008-12-1560,759 209,854 97% 4 2008-12-16 -- 2008-12-2262,527 272,381 93% 5 2008-12-23 -- 2008-12-2968,375 340,756 97% 6 2008-12-30 -- 2009-01-0573,585 414,341 97% 7 2009-01-06 -- 2009-01-1294,166 508,507 103% 8 2009-01-13 -- 2009-01-1985,557 594,064 106% 9 2009-01-20 -- 2009-01-2687,678 681,742 107% 10 2009-01-27 -- 2009-02-0291,014 772,756 110% 11 2009-02-03 -- 2009-02-0995,238 867,994 113% 12 2009-02-10 -- 2009-02-1695,967 963,961 115% 13 2009-02-17 -- 2009-02-23109,800 1,073,761 115% 14 2009-02-24 -- 2009-03-0285,246 1,159,007 -- 15 2009-03-03 -- 2009-03-09100,610 1,259,617 -- 16 2009-03-10 -- 2009-03-16100,323 1,359,940 -- 17 2009-03-17 -- 2009-03-23100,819 1,460,759 -- 18 2009-03-24 -- 2009-03-30102,843 1,563,602 -- 19 2009-03-31 -- 2009-04-06101,978 1,665,580 136% 20 2009-04-07 -- 2009-04-1399,586 1,765,166 -- 21 2009-04-14 -- 2009-04-20101,808 1,866,974 -- 22 2009-04-21 -- 2009-04-27100,230 1,967,177 -- 23 2009-04-28 -- 2009-05-0497,584 2,064,761 -- 24 2009-05-05 -- 2009-05-1195,923 2,160,684 137% 25 2009-05-12 -- 2009-05-1895,632 2,256,316 -- 26 2009-05-19 -- 2009-05-2592,377 2,348,693 -- 27 2009-05-26 -- 2009-06-0191,747 2,440,440 -- 28 2009-06-02 -- 2009-06-0891,513 2,531,953 -- Direct downloads Source: wiki/Legacy_statistics The following table shows the number of direct downloads of Fedora 10 media from unique IP addresses, as shown in the web proxy logs. The actual number of raw downloads tends to be much higher. Week Dates Downloads this week Total downloads 1 2008-11-25 -- 2008-12-01 236,886 236,886 2 2008-12-02 -- 2008-12-08 105,994 342,880 3 2008-12-09 -- 2008-12-15 83,740 426,620 4 2008-12-16 -- 2008-12-22 76,982 503,602 5 2008-12-23 -- 2008-12-29 66,351 569,953 6 2008-12-30 -- 2009-01[...]

FUDCon Toronto 2009
Another FUDCon is in the books, this time in Toronto. It was great to catch up with many people, put faces to some names, and meet a bunch of new contributors. I gave a session on Moksha, which I'll talk about below, and was also on the Fedora Infrastructure panel discussion. My goal this FUDCon wasn't to crank out a ton of code, but to focus on gathering and prioritizing requirements and to help others be productive. Here are some of the projects I focused on. Moksha Moksha is a project I created a little over a year ago, which is the base of a couple of other applications I've been working on as well: Fedora Community and CIVX. I'll be blogging about these in more detail later. One of the main themes of FUDCon this year was Messaging (AMQP), and Moksha is a large part of this puzzle, as it allows you to wield AMQP within web applications. During my session the demo involved busting open a terminal, creating a consumer that reacts to all messages, creating a message producer, and then creating a live chat widget -- all of which hooked up to Fedora's AMQP broker. I'll be turning my slides into an article, so expect a full blog post explaining the basics soon. In the mean time, I found Adam Miller's description to be extremely amusing: "I walked into a session called "Moksha and Fedora Community -- Real-time web apps with Python and AMQP" which blew my mind. This is Web3.0 (not by definition, but that's what I'm calling it), Luke Macken and J5 completely just stepped over web2.0 and said "pffft, childs play" (well not really but in my mind I assume it went something like that). This session showed off technology that allows real time message passing in a web browser as well as "native" support for standard protocols. The project page is and I think everyone on the planet should take some time to go there and enjoy the demo, prepare to have your mind blown. Oh, and I also irc transcribed that one as well ... presentation slides found:" Fedora Community So after we released v1.0 of Fedora Community for F12, all of us went off in seperate directions to hack on various things. J5 wrote AMQP javascript bindings, which I then integrated into Moksha. Máirín Duffy built a portable usability lab and has been doing great research on the usability of the project. And I dove back into Moksha to solidify the platform. After we deploy our AMQP broker for Fedora, and once we have start adding shims into our existing infrastructure, we'll then be able to start creating live widgets and message consumers that can react to events, allowing us to wield Fedora in real-time. This will let us to keep our fingers on the pulse of Fedora, automate and facilitate tedious tasks, and gather metrics as things happen. During the hackfests I also did some work on our current Fedora Community deployment. Over the past few weeks some of our widgets randomly died, and we haven't been receiving proper error messages. So, I successfully hooked up WebError and the team is now getting traceback emails, which will help us fix problems much faster (or at least nag the hell out of us about them). I also worked with Ian Weller on the new Statistics section of the dashboard, which has yet to hit production. Ian and I wrote Wiki metrics, Seth Vidal wrote BitTorrent metrics, and I wrote Bodhi metrics. We've also got many more to c[...]

TurboGears2 in Fedora & EPEL

I'm excited to announce that the TurboGears2 web application stack is now available in Fedora 12, 11 and EPEL-5.

What is TurboGears2?

TurboGears 2 is the built on top of the experience of several next generation web frameworks including TurboGears 1 (of course), Django, and Rails. All of these frameworks had limitations which were frustrating in various ways, and TG2 is an answer to that frustration. We wanted something that had:
  • Real multi-database support
  • Horizontal data partitioning (sharding)
  • Support for a variety of JavaScript toolkits, and new widget system to make building ajax heavy apps easier
  • Support for multiple data-exchange formats.
  • Built in extensibility via standard WSGI components

Installing the TurboGears2 stack & development tools

Fedora 12
yum install TurboGears2 python-tg-devtools
Fedora 11
yum --enablerepo=updates-testing install TurboGears2 python-tg-devtools
Red Hat Enterprise Linux 5 (with EPEL)
yum --enablerepo=epel-testing install TurboGears2 python-tg-devtools

Creating your first TG2 app

paster quickstart

Run your test suite


Run your application

paster serve development.ini

Read the documentation


If you're interested in helping maintain and improve the TG2/Pylons stack within Fedora/EPEL, please let me know. We're always looking for new Python hackers to join the team. There are still a few more components that need to be packaged and reviewed (eg: chameleon.genshi), so please take a look at the TurboGears2 page on the Fedora wiki for more details..

Fedora 12 is here!


Install it with the liveusb-creator!


New liveusb-creator release!

(image) So I've gotten some pretty inspiring feedback from various users of the liveusb-creator recently, so I decided to put some cycles into it this weekend and crank out another release.

"As a non-Linux person, Live-USB Creator has improved the quality of my life measurably!" --Dr. Arthur B. Hunkins
Yesterday I released version 3.8.6 of the liveusb-creator. Changes in this release include:
  • Added the F12 beta release
  • Updated to the latest Sugar on a Stick v2 beta snapshot (#522240)
  • Made our automatic device detection code more robust (#519134)
  • Fixed encoding of unicode strings from exceptions (#471367)
  • Made our Linux device detection more robust (#517053)
  • Intel Mac EFI directory preparation (#526825) thanks to Matt Domsch
  • Made our windows device detection more robust
  • Added a --device-checksum options, which calculates the checksum of the entire device.
  • Added a --liveos-checksum option, which takes the checksum of all LiveOS files, and then generate a checksum of the checksums
  • Added a --hash option for configuring the hash for the above checksum features
  • Made the LiveUSBCreator.bootable_partition method a little more robust
  • Better handling of file descriptors
  • Some Windows-specific optimizations & fixes
  • Fixed a bug with the overlay size on sticks with not much free space
  • Handle device paths containing spaces when running extlinux (#490843)
  • Remove some duplicate po files (#516841)
  • Many translation updates




Good Python Habits: vim + pyflakes

Here is a neat little hack for running pyflakes on Python files after you save them. I like using pyflakes for quickly catching dumb errors, but you could easily replace it with a more comprehensive tool like pychecker, or pylint for more strict PEP8 compliance.

All you have to do is throw this in your ~/.vimrc

au BufWritePost *.py !pyflakes %

This has saved me *tons* of time and frustration over the past few weeks, and I have no idea I lived without it.

Fedora 12 filesystem showdown