Subscribe: l e w k . o r g
http://lewk.org/blog/index.rss20
Added By: Feedage Forager Feedage Grade B rated
Language: English
Tags:
>>> quantumrandom  bodhi  data  dev  fedmsg  fedora  new  packages  quantumrandom  release  rngtest fips  rngtest  updates 
Rate this Feed
Rate this feedRate this feedRate this feedRate this feedRate this feed
Rate this feed 1 starRate this feed 2 starRate this feed 3 starRate this feed 4 starRate this feed 5 star

Comments (0)

Feed Details and Statistics Feed Statistics
Preview: l e w k . o r g

l e w k . o r g





Published: Tue, 30 Jun 2015 04:45 GMT

Copyright: Copyright 2007-2012 Luke Macken
 



Fedora 20 updates metrics, compared to F19.

Tue, 30 Jun 2015 04:45 GMT

Fedora 20 recently reached End Of Life. Below are metrics generated from Bodhi, along with the percent change compared to Fedora 19. 15012 updates (+8.37%) 6952 packages updated (+6.09%) 13207 stable updates (+8.59%) 250 testing updates (+8.69%) 35 pending updates (-10.26%) 1520 obsolete updates (+14.72%) 8397 bugfix updates (55.94% of updates) (+5.60%) 3605 enhancement updates (24.01% of updates) (+17.89%) 1114 security updates (7.42% of updates) (+15.80%) 1896 newpackage updates (12.63% of updates) (+6.28%) 12315 bugs resolved (+13.61%) 1603 critical path updates (10.68% of updates) (+44.41%) 1560 approved critical path updates (+43.12%) 0 unapproved critical path updates (+0%) 144 critical path security updates (16.78% of security updates, 14.02% of critpath updates) (-15.79%) 5455 updates received feedback (36.34% of updates) (+34.29%) 1259 +0 comments (+41.62%) 11579 +1 comments (+47.79%) 678 -1 comments (+29.14%) 1195 unique authenticated karma submitters (+15.35%) 49 proventesters (+2.08%) 1681 +1's from proventesters (+5.13%) 96 -1's from proventesters (+41.18%) 5 critpath updates with conflicting proventesters (0.31% of critpath updates) (+25%) dnf-0.4.18-1.fc20 kernel-3.13.4-200.fc20 perl-5.18.4-290.fc20 sudo-1.8.11p2-1.fc20 testdisk-6.14-3.fc20 15 critpath updates with positive karma and negative proventester feedback (0.94% of critpath updates) (+200%) createrepo-0.9.9-23.fc20 dnf-0.4.15-1.fc20 dnf-0.4.18-1.fc20 dracut-037-10.git20140402.fc20 gnome-shell-3.10.4-6.fc20 kernel-3.11.10-301.fc20 kernel-3.13.3-201.fc20 kernel-3.13.4-200.fc20 kernel-3.14.2-200.fc20 libsolv-0.6.0-0.git05baf54.fc20 llvm-3.4-10.fc20 PackageKit-0.8.14-2.fc20 perl-5.18.4-290.fc20 sudo-1.8.11p2-1.fc20 testdisk-6.14-3.fc20 630 critpath updates with positive karma and positive proventester feedback (39.30% of critpath) (+36.07%) 304 anonymous users gave feedback (2.26%) (+49.75%) 2396 updates reached the stable karma threshold (18.14%) (+36.60%) 11200 updates reached the minimum time in testing threshold (84.80%) (+1.91%) 8259 went from testing to stable *without* karma (63.79%) (-3.79%) 33 updates were pushed to stable with negative karma (0.25%) (-10.81%) 57 critical path updates pushed to stable *without* karma (-60.96%) Time spent in testing: mean = 13 days (+8.33%) median = 9 days (+12.5%) mode = 8 days (+0%) [...]



Release Tools and Infrastructure FAD

Fri, 26 Jun 2015 22:54 GMT

The Release Tools and Infrastructure Fedora Activity Day happened recently at the Red Hat office in Westford, Massachusetts. The goal was to bring our release tooling and processes up to speed with the current and future demands of the Fedora Project. Since there are a ton of moving parts of the Fedora Release Engineering community that need work, many of us split out into groups to tackle various components. Even though there were a ton of shiny tasks that I would have loved to dive into, like pungi4, createrepo_c, koji 2.0, etc, I felt like my cycles were best spent working on Bodhi 2.0. Bodhi is a critical piece of infrastructure that has been in production for over 7 years now, and it's initial incarnation was deployed internally to Red Hat for Fedora Core prior to that. It is currently a cross-section between releng, qa, and infra, and is responsible for shipping updated bits to users while facilitating community feedback. It started with pushing out updated RPMs initially, but since F21 it has also started updating Atomic OSTrees as well. With the number of requirements for updated deliverables growing each release, it's clear that our current architecture does not scale with this appropriately. The next generation of Bodhi has been in the works for a while now, and is almost ready for testing in staging. So I started off the FAD by getting the current bodhi2 stack built for RHEL7. This turned out to be a non-trivial task, as I had to build a bunch of complex dependencies like python-cryptography, python-six, and python-py/pytest (which have circular deps on eachother). Thankfully, I now have a bodhi2 copr setup for building the extra pieces of the stack until they are available in RHEL7+EPEL. I then began writing a new bodhi2 ansible role and got an instance deployed to staging. There is still a bunch of work to be done before the staging instance is usable, but that mostly revolves around configuration and hopefully shouldn't be too far off. Once we have something we can start playing with, there is going to be a bunch of work needed to port the existing tools over to use the new API. I'm hoping we can keep the python-fedora bindings "stable", but there are still plenty of other things that hit the JSON API directly that will most likely need tweaking. As far as future goals for the Bodhi architecture goes, I'd really like to see us split it up into a couple of different components. Ideally, I think the majority of it would revolve around the web frontend & RESTful JSON API that facilitates community testing. There's already a pretty awesome ecosystem that has been built around this, with client-side QA tools like fedora-easy-karma and fedora-gooey-karma which make it easy people test and give feedback to things in updates-testing that they have installed. Then there is "The Masher" backend, which currently does most of the heavy lifting. Most of this could ideally be pushed into pungi4 and koji plugins, for things like mashing updates repos, composing ostrees, generating updated images/installers, etc. This way, bodhi would simply handle tagging things in koji and then orchestrate the creation of the updated bits with pungi4, update/close bugs, generate updateinfo metadata, etc. Then we can finally leverage our build farm to perform the long-running composition tasks in parallel, instead of doing them all linearly on a single releng machine. So bodhi would still need to handle the orchestration of these processes, but I could also see us pushing this stuff into the theoretical "ComposeDB" idea that we have been tossing around as well. This could then potentially take care of rebuilding/composing whatever needs to be updated based on what has changed. I wasn't directly involved with the detailed requirements discussions between Dennis, Ralph, etc.. but from my perspective I am really excited about the idea of a central tool that knows about all release artifacts, what is contained within them, and how to produce them. Then when something like a critical security u[...]



Atomic OSTree updates for Fedora 21

Sat, 10 Jan 2015 00:12 GMT

With the new Fedora 21 Cloud Atomic Host released, we needed a way to continuously update the OSTree during the Fedora updates process. The rawhide & branched nightly tree composes are triggered by cron jobs, but we needed to somehow tie it into the Bodhi push process for F21. There are a number of ways to compose OSTrees, but for this task we needed a lot more control and flexibility than things like the rpm-ostree-toolbox give us. So I wrote a new tool called (for the lack of a better name) the fedmsg-atomic-composer. It is comprised of a command-line tool, a Python API, as well as a fedmsg consumer that can trigger tree composes whenever Fedora repositories are updated. Here's how it works: The default config.py contains all of the release information, paths and commands, which can easily be modified to customize your tree. It starts by rsyncing the canonical production OSTree repository locally, if one exists. The corresponding fedora-atomic.git branch is cloned, which contains the parent fedora-atomic-docker-host.json treefile that we inherit from. We then dynamically generate the mock configuration and yum repo files from Mako templates based on the config.py values. The mock chroot is then initialized with the latest OSTree stack from our repositories. With the latest version of mock, it will use a systemd-nspawn container instead of a chroot. The OSTree is then composed in the mock container with our custom treefile Upon completion, the OSTree summary file is updated. This is a serialized GVariant that describes the available branches along with other metadata. During the recent MirrorManager 2 FAD, Ralph implemented native support for OSTrees, which relies on this file. The updated tree is then synced back out to production, where users will then be able to atomic upgrade and pull in the changes. From here, hooking this tool into the Bodhi push process was fairly simple. All we needed to do was to update the repo URLs in our config.py to point to our freshly mashed local repositories, and then we're good to go. So far the tree composes has been churning along smoothly, and we have yet to encounter any major issues. I think the next step for our Atomic infrastructure is probably to get the new MirrorManager2 deployed so we can start to balance the load a bit. [...]



liveusb-creator 3.13 release!

Sun, 07 Dec 2014 23:30 GMT

A new version of the liveusb-creator has just hit the Fedora repos! It's been a long time since there has been any major changes to the tool, but I managed to find a couple of days last week to implement some much-needed functionality.

(image)

This version features a new destructive installation method, which uses dd to directly copy the the entire ISO to to your device, overwriting your existing data and partitions. This method tends to be much more reliable, since we don't have to crack open the ISOs, mess with partition flags, or configure and install the bootloader ourselves. This also means that DVD, netinst, and boot images finally work with this new method as well. You can still perform the traditional non-destructive install that the liveusb-creator has always done, and it remains the default.

Unfortunately there is not a Windows build for this release just yet, as I hit some issues getting dd.exe to work properly in my virtual machine. Hopefully I'll be able to figure it out at some point next week. If any Windows folks are out there and want to help out, checkout the Developers Guide for instructions on setting things up.

There are also a number of other features and fixes in this release, checkout the wiki for more details. The source code can also be found on GitHub.




Bodhi2 Fedora Activity Day

Mon, 16 Jun 2014 16:49 GMT

The Bodhi2/Taskotron Fedora Activity Day happened earlier this month! A bunch of us gathered in Denver for a few days and worked on some of our critical releng & qa infrastructure. The hackfest was held in a conference room in my apartment building, which worked out quite nicely for the amount of people that we had. The hotel was right up the road, and we were able to walk to a lot of awesome spots, like the 1UP Barcade :). It was great to have folks from various corners of Fedora together in the same room for a few days. As it is, we get a lot done remotely, but being able to leverage the high-bandwidth face-to-face time is extremely valuable, especially when coming to consensus on difficult architectural decisions. We used Gobby to collaborate on a long list of action items, and chipped away most of it. Thankfully, Bodhi has enough layers that we were all able to split up and dive into different corners of the code without stepping on each others' toes. Up until now, our releng stack in staging has always been less than useful. We've never able to do a full build->update->push->run, and have had to rely on testing certain codepaths in production. Not only that, but Bodhi's "masher" never really had proper unit tests, so pushing out major changes to that stack has always been quite unpleasant. Thankfully, Kevin and Dennis worked on our staging setup and made it so we can actually use it. I made a lot of headway on porting the Bodhi masher to a new fedmsg-driven architecture, while writing unit tests for it along the way. I'm hopeful that we can write tests for every part of the "push" process, and then optimize the hell out of it :) While Mathieu and I mainly focused on back-end hacking, Ralph and Ricky made some fantastic headway on the front-end. Ralph started working on the revamped New Update Form, which is already significantly better than the original. The idea here is that the maintainer should only need to provide the package name(s), and Bodhi will automatically find the corresponding candidate builds, related bugs, and eventually it will pull in all candidate deps as well (or tell if you if any need to be rebuilt). It would also be very convenient to be able to "push this entire stack again". Ideally, I'd love to make it so that folks maintaining large stacks, like GNOME, shouldn't need to use a Google doc to coordinate their mega-updates ;) Ralph also started revamping the new karma system (check out his screencast here). We don't have any of the policy in place to enforce it yet, but eventually we'd like the maintainers to be able to define custom policy constraints for their updates. For example, you could say "only allow this update to go to the stable repo once this specific bug or test case has been confirmed as fixed/passing". Ricky made lots of improvements to the Release profiles and Updates/Comments/User pages, which are all looking great. He also created a Bodhi news feed on the front page using the fedmsg datagrepper widget that Ralph blogged about recently. Other front-end improvements include libravatar support for all users, proper markdown rendering with previews and image support, and of course a konami code easter-egg ;) I was going to post a bunch of screenshots here, but Ralph just deployed a development instance of Bodhi2 that folks can play around with instead: http://bodhi.dev.fedoraproject.org (it's a dev instance, so don't expect it to always be up/functional). Some other corners of Bodhi that have seen improvements recently: The API. The Bodhi webapp is written in Python using the excellent Pyramid web framework. On top of that we are using Cornice, which makes it really easy to build & document standards-compliant web services with Pyramid. Thanks to colander validation/deserialization schemas and our custom ACLs and validators, we are able to write dead-simple views that can safely assume that all of the data we are dealing with is valid[...]



Keeping your finger on the pulse of the Fedora community

Thu, 07 Mar 2013 17:30 GMT

For those who haven't been keeping up with all of the awesome code Ralph Bean has been churning out lately, be sure to checkout fedmsg.com. Hop on #fedora-fedmsg on Freenode or load up busmon to see it in action. Not all of the Fedora Infrastructure services currently fire off fedmsgs, but we're getting very close. This technology is built on top of Moksha, which I created many years ago while writing the first version of the fedoracommunity app. It's come a long way since then, and now can speak ØMQ over WebSockets, as well as AMQP and STOMP over Orbited. Now the time has finally come to bring Moksha to the desktop! Introducing fedmsg-notify fedmsg-notify lets you get realtime desktop notifications of activity within the Fedora community. It allows you to tap into the firehose of contributions as they happen and filter them to your liking. It works with any window manager that supports the notification-spec, however I've only seen the gravatars show up using GNOME. For GNOME Shell users, you can [optionally] install gnome-shell-extension-fedmsg, and then enable it with the gnome-tweak-tool or by running `gnome-shell-extension-tool -e fedmsg@lmacken-redhat.com` (and then hit alt+f2 and type 'r' to reload the shell). You will then be graced with the presence of The Bus: For those who aren't running GNOME shell, you can simply yum install fedmsg-notify, and then run fedmsg-notify-config, or launch it from your Settings menu. Due to a dependency on Twisted's gtk3reactor, fedmsg-notify is currently only available on Fedora 18 and newer. The first tab shows you all services that are currently hooked into fedmsg. As we add new ones, the gui will automatically display them. These services are defined in the fedmsg_meta_fedora_infrastructure package. The Advanced tab lets you further customize what messages you want to see. The "Bugs that you have encountered" option will display all messages that reference any Bugzilla numbers for crashes that you have hit locally with ABRT. The other filters involve querying your local yum database or the PackageDB. Under the hood The fedmsg-notify-daemon itself is fairly minimal (see daemon.py). At it's core, it's just a Twisted reactor that consumes ØMQ messages. Moksha does all of the heavy lifting behind the scenes, so all we really have to do is specify a topic to subscribe to and define a consume method that gets called with each message. This is essentially just a basic Moksha Consumer with some fedmsg + DBus glue. class FedmsgNotifyService(dbus.service.Object, fedmsg.consumers.FedmsgConsumer): topic = 'org.fedoraproject.*' def consume(self, msg): … The daemon will automatically startup upon login, or will get activated by DBus when enabled via the GUI. When a message arrives, it filters it accordingly, downloads & caches the icons, [optionally] relays the message over DBus, and then displays the notification on your desktop. The API for writing custom filters is dead simple (see filters.py). Here is an example of one: class MyPackageFilter(Filter): """ Matches messages regarding packages that a given user has ACLs on """ __description__ = 'Packages that these users maintain' __user_entry__ = 'Usernames' def __init__(self, settings): self.usernames = settings.replace(',', ' ').split() self.packages = set() reactor.callInThread(self._query_pkgdb) def _query_pkgdb(self): for username in self.usernames: log.info("Querying the PackageDB for %s's packages" % username) for pkg in PackageDB().user_packages(username)['pkgs']: self.packages.add(pkg['name']) def match(self, msg, processor): packages = processor.packages(msg) for package in self.packages: if package in packages: [...]



Wielding the ANU Quantum Random Number Generator

Sat, 21 Apr 2012 16:30 GMT

Last week Science Daily published an article that caught my attention titled 'Sounds of Silence' Proving a Hit: World's Fastest Random Number Generator. The tl;dr is that researchers at the ANU ARC Centre of Excellence for Quantum Computation and Communication Technology created a blazing fast random number generator based on quantum fluctuations in a vacuum. Thankfully, these awesome scientists are giving their data away for free, and they even provide a JSON API. In an effort to make it simple to leverage this data, I created a new project: quantumrandom. It provides a qrandom command-line tool, a Python API, and also a /dev/qrandom Linux character device. Installing $ virtualenv env $ source env/bin/activate $ pip install quantumrandom Using the command-line tool $ qrandom --int --min 5 --max 15 7 $ qrandom --binary ���I�%��e(�1��c��Ee�4�������j�Կ��=�^H�c�u oq��G��Z�^���fK�0_��h��s�b��AE=�rR~���(�^ +a�a̙�IB�,S�!ꀔd�2H~�X�Z����R��.f ... $ qrandom --hex 1dc59fde43b5045120453186d45653dd455bd8e6fc7d8c591f0018fa9261ab2835eb210e8 e267cf35a54c02ce2a93b3ec448c4c7aa84fdedb61c7b0d87c9e7acf8e9fdadc8d68bcaa5a ... Creating /dev/qrandom quantumrandom comes equipped with a multi-threaded character device in userspace. When read from, this device fires up a bunch of threads to fetch data. Not only can you utilize this as a rng, but you can also feed this data back into your system's entropy pool. In order to build it's dependencies, you'll need the following packages installed: svn gcc-c++ fuse-devel gccxml libattr-devel. On Fedora 17 and newer, you'll also need the kernel-modules-extra package installed for the cuse module. pip install ctypeslib hg+https://cusepy.googlecode.com/hg sudo modprobe cuse sudo chmod 666 /dev/cuse qrandom-dev -v sudo chmod 666 /dev/qrandom By default it will use 3 threads, which can be changed by passing '-t #' into the qrandom-dev. Testing the randomness for FIPS 140-2 compliance $ cat /dev/qrandom | rngtest --blockcount=1000 rngtest: bits received from input: 20000032 rngtest: FIPS 140-2 successes: 1000 rngtest: FIPS 140-2 failures: 0 rngtest: FIPS 140-2(2001-10-10) Monobit: 0 rngtest: FIPS 140-2(2001-10-10) Poker: 0 rngtest: FIPS 140-2(2001-10-10) Runs: 0 rngtest: FIPS 140-2(2001-10-10) Long run: 0 rngtest: FIPS 140-2(2001-10-10) Continuous run: 0 rngtest: input channel speed: (min=17.696; avg=386.711; max=4882812.500)Kibits/s rngtest: FIPS tests speed: (min=10.949; avg=94.538; max=161.640)Mibits/s rngtest: Program run time: 50708319 microseconds Adding entropy to the Linux random number generator sudo rngd --rng-device=/dev/qrandom --random-device=/dev/random --timeout=5 --foreground Monitoring your available entropy levels watch -n 1 cat /proc/sys/kernel/random/entropy_avail Python API The quantumrandom Python module contains a low-level get_data function, which is modelled after the ANU Quantum Random Number Generator's JSON API. It returns variable-length lists of either uint16 or hex16 data. >>> quantumrandom.get_data(data_type='uint16', array_length=5) [42796, 32457, 9242, 11316, 21078] >>> quantumrandom.get_data(data_type='hex16', array_length=5, block_size=2) ['f1d5', '0eb3', '1119', '7cfd', '64ce'] Based on this get_data function, quantumrandom also provides a bunch of higher-level helper functions that make easy to perform a variety of tasks. >>> quantumrandom.randint(0, 20) 5 >>> quantumrandom.hex()[:10] '8272613343' >>> quantumrandom.binary()[0] '\xa5' >>> len(quantumrandom.binary()) 10000 >>> quantumrandom.uint16() numpy.array([24094, 13944, 22109, 22908, 34878, 33797, 47221, 21485, 37930, ...], dtype=numpy.uint16) >>> quantumrandom.uint16().data[:10] '\x87\x7fY.\xcc\xab\xea\r\x1c`' Follow quantumrandom on GitHub: https://github.com/lmacken/q[...]