Last Build Date: Wed, 10 Aug 2016 20:34:24 GMT
Wed, 10 Aug 2016 20:34:24 GMTThe annual Fedora contributor's conference, Flock to Fedora, wrapped up last week. From everything I've seen and heard, it was a smashing success. The trip reports and writeups have been very detailed and helpful, so I thought I would take a slightly different perspective with mine. I'll cover a few sessions, but most of this will be from an organizers viewpoint.However, the highlight for me was meeting three of my new team members in person (or in the case of Dan, seeing him again). My transition to the team had just completed and it was almost purely coincidence that all of us were going to be in the same place at the same time. I appreciate the opportunity to see them, as I really value face to face time with a team. It was a great way to start off on a good note. Sessions The Kernel Talk For the first time in several years, someone else gave the Fedora kernel talk. Laura Abbott did a fantastic job with this. I thought she struck a good balance in using statistics to emphasize her points where needed, whereas my talks tended to be heavy on statistics for statistics sake. The overall theme of inclusion and community during the talk was also excellent. We need the community around the Fedora kernel to grow, and Laura made that point numerous times. Hopefully people watch the video when it is online and feel empowered to reach out to the kernel maintainers. I promise they are very approachable people. Ensuring ABI stability in Fedora Dodji Seketeli and Sinny Kumari gave a fascinating talk on abicheck and libabigail, and it's potential uses in Fedora. Having done work on toolchains and ELF in the distant past, I was very impressed on how they are able to present an ABI difference between two package versions down to the individual .h file level and present that in a format that humans are comfortable reading. Hopefully as work continues, we'll be able to utilize this in the Fedora updates process to detect inadvertent ABI changes before they land on users systems. I also think it could be useful in the Modularity effort to validate that the API/ABI a module claims to export is actually what is exported. I'm definitely going to keep an eye on the work Dodji and Sinny are doing here. Fedora Council Townhall and Update This session was kind of a bust for it's intended purpose. The people in the room were literally the Council members, a couple of members of OSAS, and one or two people that I think were really just looking for a quiet room to hack in. However, not to make it a waste, we spent most of the session discussing some Flock itself among the Council and OSAS members. I felt this was fairly productive, but there wasn't anything firm enough to recap here. Hopefully the results of some of that discussion are posted in the not distant future. Modularity I've seen Langdon give his Modularity talk 3 or 4 times now across various venues. Every time I watch it, I'm amazed at its ambition and relevance in today's trending software market. The nice thing about this session is that it continues to show progress towards actually doing this every time. This particular time around, Langdon was able to show off a few demos of installing various modules (a kernel and httpd). Live demos are always fun, and this was no exception. I think his team is doing great work, and I'm looking forward to seeing how this plays into the way we create Fedora in the future. There are a lot of details to work out and that's where the fun part will be. PRD Workshop I wound up, somewhat unexpectedly, moderating a PRD workshop for the Fedora Editions. The PRDs for each WG have existed for a few releases now, and some of them are woefully out of date. It was time to refresh what each WG was aiming for, and document that.Rather than dig into the existing PRDs and revise from there, we started off by using the Kellogg Logic Model that Matthew Miller covered in his talk earlier in the day. We started with Impact. So we listed Mission and Vision statements, followed by Outcomes that map to those, followed by Outputs that would [...]
Wed, 10 Aug 2016 15:00:03 GMTI've been doing kernel development or maintenance for a large portion of my professional career. It started at my previous employer and continued for the past 5 years at Red Hat on the Fedora kernel team, the last 3 as the defacto team lead. It has been a fantastic experience, from both technical and people perspectives. I consider myself extremely lucky to have been a part of the Fedora kernel team, both past and present incarnations of it. Which is why when career discussions came up recently, it wasn't the easiest thing to think about.
Thu, 30 Jun 2016 16:27:20 GMTThink of this scenario: You're sitting at your shiny Fedora install and notice a kernel update is available. You get all excited, update it through dnf or Gnome Software, or whatever you use, reboot and then things stop working. "STUPID KERNEL UPDATE WHY DID YOU BREAK MY MACHINE" you might say. Clearly it's at fault, so you dutifully file a bug against the kernel (you do that instead of just complaining, right?). Then you get told it isn't a kernel problem, and you probably think the developers are crazy. How can a kernel update that doesn't work NOT be a kernel problem?This scenario happens quite often. To be sure, a good portion of the issues people run into with kernel updates are clearly kernel bugs. However, there is a whole set of situations where it seems that way but really it isn't. So what is at fault? Lots of stuff. How? Let's talk about the boot sequence a bit. Booting: a primer Booting a computer is a terribly hacky thing. If you want a really deep understanding of how it works, you should probably talk to Peter Jones. For the purposes of this discussion, we're going to skip all the weird and crufty stuff that happens before grub is started and just call it black magic.Essentially there are 3 main pieces of software that are responsible for getting your machine from power-on to whatever state userspace is supposed to be in. Those are grub, the kernel, and the initramfs. Grub loads the kernel and initramfs into memory, then hands off control to the kernel. The kernel does the remainder of the hardware and low-level subsystem init, uncompresses the initramfs and jumps into the userspace code contained within. The initramfs bootstraps userspace as it sees fit, mounting the rootfs and switching control to that to finish up the boot sequence. Seems simple. The initramfs So what is this "initramfs"? In technical terms, it's a weird version of a CPIO archive that contains a subset of userspace binaries needed to get you to the rootfs. I say weird because it can also have CPU microcode tacked onto the front of it, which the kernel strips off before unpacking it and applies during the early microcode update. This is a good thing, but it's also kind of odd.The binaries contained within the initramfs are typically your init process (systemd), system libraries, kernel modules needed for your hardware (though not all of them), firmware files, udev, dbus, etc. It's almost equivalent to the bare minimum you can get to a prompt with. If you want to inspect the contents for yourself, the lsinitrd command is very handy.There are actually a couple of different 'flavors' of initramfs as well. The initramfs found in the install images is a generic initramfs that has content which should work on the widest variety of machines possible, and can be used as a rescue mechanism. It tends to be large though, which is why after an install the initramfs is switched to HostOnly mode. That means it is specific to the machine it is created on. The tool that creates the initramfs is called dracut, and if you're interested in how it works I would suggest reading the documentation. The problems OK, so now that we have the components involved, let's get to the actual problems that look like kernel bugs but aren't. Cannot mount rootfs One of the more common issues we see after an update is that the kernel cannot mount the rootfs, which results in the system panicking. How does this happen? Actually, there are a number of different ways. A few are:* The initramfs wasn't included in the grub config file for unknown reasons and therefore wasn't loaded.* The initramfs was corrupted on install.* The kernel command line specified in the grub config file didn't include the proper rootfs arguments.All of those happen, and none of them are the kernel's fault. Fortunately, they tend to be fairly easy to repair but it is certainly misleading to a user when they see the kernel panic. A different update breaks your kernel update We've established that the initramfs is a [...]
Wed, 11 May 2016 18:03:56 GMTOften we get bugs reported against the Fedora kernel for issues involving third party drivers. Sometimes those are virtualbox, sometimes VMWare guest tools, but most often it is the nvidia driver. We had another reported today. I'll pause to let you read it. Go ahead, read the whole thing. I'll wait.https://bugzilla.redhat.com/show_bug.cgi?id=1335173Done? Good. You'll notice a couple things. First, it's closed as CANTFIX with a rote comment that we typically use for such bugs. That comment, while terse, is not incorrect. Second, the reporter is, frankly, pissed off. You know what?He has every right to be.So if the response in the bug is not incorrect, how does that line up with the assertion that the reporter's anger isn't wrong either? I thought I'd spend some time breaking this bug down in detail to try and explain it.The crux of the reporters argument is that using the nvidia driver on Fedora causes pain to Fedora's users. I'm not going to argue against that. Using the nvidia driver on Fedora is very much painful. A user can finally get it working, and then we rebase the kernel and it breaks again for them. That isn't a good user experience at all. We've know this for a while and there are some tentative plans to help users in this situation by defaulting to a known working kernel if they have the nvidia driver installed. That doesn't fix the problem, but it at least reduces the element of surprise.The reporter then goes on to make some assertions that might seem plausible, but in fact aren't accurate at all. Let's look at these more closely. The claims You do not intentionally break hardware compatibility? Oh, wait, you do.We do not intentionally break anything. As we've written about before, we rebase the kernel to pick up the bugfixes the upstream maintainers are including in those newer releases. However, an additional benefit of those rebases is that we actively enable more new hardware by doing so. Yes, there are regressions and they are particularly prevalent if you are relying on out-of-tree drivers. Those regressions are unfortunate, but certainly not done out of malice.You do not intentionally break API/ABI compatibility? Oh, wait, you do.Greg-kh has talked and written extensively about the fact that the upstream kernel has no stable API. In fact, his document is included in the kernel source itself. Due to the fact that the kernel has no stable API, there is also no stable ABI. Now, it should be noted that ABI here is describing the ABI between the kernel and modules, not the ABI between the kernel and userspace. The kernel/userspace ABI is done via syscalls and that is stable and fanatically protected by the upstream kernel maintainers. However, modules don't have that luxury and therefore when a rebase is done the ABI can and does change. Include the fact that compiler versions change in Fedora, which can impact ABI, and it becomes evident that the reporter's claim is somewhat true.We could freeze on a kernel version and attempt to keep the API/ABI stable, but that incurs a significant maintenance cost that our small team is not able to handle. Even the RHEL kernel, with it's much larger user base and development team, has a limited kABI they support. So yes, the API/ABI changes with a rebase (or more rarely with a new stable update), but that is one of the consequences of doing a rebase. It is not done with the intention of breaking anything purposefully.You do not limit user's choice in regard to running software or drivers? Oh, wait, you do.Fedora actually does not limit the user's choice in software. The user is free to install the nvidia driver or whatever other software they wish to use. Google Chrome, Lotus Notes, Steam, nvidia drivers, and other software not provided by Fedora has all been known to be installed and work. There is no problem with a user choosing to do this. It is their own computer!What the Fedora kernel team cannot do is provide support for such software. As Just[...]
Fri, 04 Mar 2016 19:51:45 GMTTLDR: Update xfsprogs to xfsprogs-4.3.0-1 before you update the kernel to 4.4.4 on Fedora 22/23.Every once in a while we get a bug report that sounds really really bad. So bad that it makes us do a double take and start thinking about it in detail even before we complete the normal daily bug review. Today was one of those days.This morning I read a bug report that suggested the 4.4 kernel will not boot with an xfsprogs in userspace less than version 4.3.0. That sounds pretty bad. Fedora 23 has already been rebased to the 4.4.y stable series of kernels, and it is in update-testing for Fedora 22. Given that the default filesystem in Server edition is XFS, that definitely caused some concern.However, thinking about it even for a moment it seemed somewhat odd. First, typically the only userspace component involved in mounting an XFS filesystem is mount(8). That isn't provided by the xfsprogs package. Even if that is what was broken, one would think that there would have been bug reports long before now given that Fedora 23 was rebased a while ago already. Perhaps fsck might come into play too, and that's a possibility but it isn't very likely. So it was time to dig in further. The report linked to an upstream discussion. Here is where things started to get clearer. I'll try and explain a bit.Around the 3.15 kernel release, XFS added support for maintaining CRC checksums for on-disk metadata objects. According to the mkfs.xfs manpage:CRCs enable enhanced error detection due to hardware issues, whilst the format changes also improves crash recovery algorithms and the ability of various tools to validate and repair metadata corruptions when they are found. The CRC algorithm used is CRC32c, so the overhead is dependent on CPU archi‐ tecture as some CPUs have hardware acceleration of this algorithm. Typically the overhead of calculat‐ ing and checking the CRCs is not noticable in normal operation.Yay! More robustness and better error recovery. Fantastic. However, features like this are rarely enabled by default when they are first released. In fact, the addition of this necessitated bumping the version number of the XFS on-disk format to version 5. For further caution, xfsprogs didn't start supporting this until xfsprogs-3.2.0 and didn't make it the default for mkfs.xfs until xfsprogs-3.2.3. Pretty cautious on the part of the XFS developers. But that still leaves us trying to figure out why the report was submitted and what it means for Fedora.First, the actual problem. In the 4.4 kernel release the XFS developers started validating some additional metadata of V5 filesystems against the log. This is fine for healthy filesystems. However, if you had a crash and ran xfs_repair from an older xfsprogs package, it would unconditionally zero out the log and the kernel would be very confused and report corruption when you tried to mount it. The xfsprogs-4.3 release fixed this in the xfs_repair utility. OK, so there's validity to the 4.4 kernel needing xfsprogs-4.3, but only in certain circumstances. Namely, you have to have a V5 filesystem on disk, and you have to run xfs_repair from an older xfsprogs against it.Fortunately, that isn't a super common situation. For Fedora 22, we released with xfsprogs-3.2.2. That means any new XFS filesystems created from installation media should still be V4 and not hit this issue (prior EOL Fedora releases are in the same boat). It's possible someone manually specified crc=1 when they created their F22 XFS partition, but that is a rare case. Fedora 23, on the other hand, shipped with xfsprogs-3.2.4 and should be creating V5 XFS filesystems by default. Those users would still need to run xfs_repair, which isn't a massively[...]
Thu, 03 Dec 2015 02:14:27 GMTA few days ago, a Fedora community member was asking if there was a 4.3 kernel for F23 yet (there isn't). When pressed for why, it turns out they were asking for someone that wanted newer support but thought a 4.4-rc3 kernel was too new. This was surprising to me. The assumption was being made that 4.4-rc3 was unstable or dangerous, but that 4.3 was just fine even though Fedora hasn't updated the release branches with it yet. This led me to ponder the upstream kernel release cycle a bit, and I thought I would offer some insights as to why the version number might not represent what most people think it does.First, I will start by saying that the upstream kernel development process is amazing. The rate of change for the 4.3 kernel was around 8 patches per hour, by 1600 developers, for a total of 12,131 changes over 63 days total. And that is considered a fairly calm release by kernel standards. The fact that the community continues to churn out kernels of such quality with that rate of change is very very impressive. There is actually quite a bit of background coordination that goes on between the subsystem maintainers, but I'm going to focus on how Linus' releases are formed for the sake of simplicity for now.A kernel release is broken into a set of discrete, roughly time-based chunks. The first chunk is the 2 week merge window. This is the timeframe where the subsystem maintainers send the majority of the new changes for that release to Linus. He takes them in via git pull requests, grumbles about a fair number of them, refuses a few others. Most of the pull requests are dealt with in the first week, but there are always a few late ones so Linus waits the two weeks and then "closes" the window. This culminates in the first RC release being cut.From that point on, the focus for the release is fixes. New code being taken at this point is fairly rare, but does happen in the early -rc releases. These are cut roughly every Sunday evening, making for a one week timeframe per -rc. Each -rc release tends to be smaller in new changesets than the previous, as the community becomes much more picky on what is acceptable the longer the release has gone on. Typically it gets to -rc7, but occasionally it will go to -rc8. One week after -rc7 is released, the "final" release is cut, which maps nicely with the 63 day timeframe quoted above.Now, here is where people start getting confused. They see a "final" release and immediately assume it's stable. It's not. There are bugs. Lots and lots of bugs. So why would Linus release a kernel with a lot of bugs? Because finding them all is an economy of scale. Let's step back a second into the development and see why.During the development cycle, people are constantly testing things. However, not everyone is testing the same thing. Each subsystem maintainer is often testing their own git tree for their specific subsystem. At the same time, they've opened their subsystem trees for changes for the next version of the kernel, not the one still in -rcX state. So they have new code coming in before the current code is even released. This is how they sustain that massive rate of change.Aside from subsystem trees, there is the linux-next tree. This is daily merge of all the subsystem maintainer trees that have already opened up to new code on top of whatever Linus has in his tree. A number of people are continually testing linux-next, mostly through automated bots but also in VMs and running fuzzers and such. In theory and in practice, this catches bugs before they get to Linus the next round. But it is complicated because the rate of change means that if an issue is hit, it's hard to see if it's in the new new code only found in linux-next, or if it's actually in Linus' tree. Determining that usually winds up being a manual process via git-bisect, but sometimes the testing bots can determine the offending commit in an automated fa[...]
Tue, 06 Oct 2015 17:14:00 GMTA while back I wrote about some efforts to move to using an exploded source tree for the Fedora kernel. As that post details, it wasn't the greatest experience. However, I still think an exploded tree has good utility and I didn't want to give up on the idea of it existing. So after scraping our "switch" I decided to (slowly) work on a tool that would create such a tree automatically. In the spirit of release-early and pray nobody dies from reading your terrible code, we now have fedkernel. The readme file in the repo contains a high level overview of how the tool works, so I won't duplicate that here. Instead I thought I would talk about some of the process changes and decisions we made to make this possible.Git all the thingsOne of the positive fallouts of the previous efforts was that all of the patches we carried in Fedora were nicely formatted with changelogs and authorship information. Being literally the output of git-format-patch instantly improved the patch quality. When it came time to figure out how to generate the patches from pkg-git to apply to the exploded tree, I really wanted to keep that quality. So I thought about how to accomplish this and then I realized there was no need to reinvent the wheel. The git-am and git-format-patch tools existed and were exactly what I wanted.After discussing things with the rest of the team, we switched to using git-am to apply patches in the Fedora kernel spec. The mechanics of this are pretty simple: the spec unpacks the tarball (plus any -rcX patches) and uses this as the "base" commit. Stable update patches are applied as a separate commit on top of the base if it is a stable kernel. Then it walks through every patch and applies it with git-am. This essentially enforces our patch format guidelines for us. It does have the somewhat negative side effect of slowing down the %prep section quite a bit, but in practice it hasn't been slow enough to be a pain. (Doing a git add and git commit on the full kernel sources isn't exactly speedy, even on an SSD.)So after %prep is done, the user is left with a git tree in the working directory that has all the Fedora patches as separate commits. "But wait, isn't the job done then?", you might ask. Well, no. We could call it good enough, but that isn't really what I or other users of an exploded tree were after. I wanted a tree with the full upstream commit history plus our patches. What this produces is just a blob, plus our patches. Not quite there yet but getting closer.SnakesThis is where fedkernel comes in. I needed tooling that could take the patches from this franken-tree and apply them to a real exploded git tree. My previous scripts were written in bash, and I could have done this in bash again but I wanted to make it automated and I wanted it to talk to the Fedora infrastructure. This means it was time to learn python again. Fortunately, the upstream python community has great documentation and there are modules for pretty much anything I needed. This makes my "bash keyboard in interactive python session" approach to the language pretty manageable and I was able to make decent progress.To get the patches out of the prepped sources, I needed to know mainly one thing. What was the actual upstream base for this build? That is easy enough to figure out if you can parse certain macros in kernel.spec. The one part that proved to be somewhat difficult was for git snapshot kernels. We name these -gitY kernels, where X increases until the next -rcX release. E.g. kernel-4.3.0-0.rc3.git1.1, kernel-4.3.0-0.rc3.git2.1, etc. That's great for RPMs, but the only place we actually documented what upstream commit we generated the snapshot from was in an RPM %changelog comment.Parsing it out of there is possible, but it's a bit cumbersome and it is somewhat error prone. The sha1sum is always recorded, but it isn't guaranteed to be the newest change[...]
Wed, 09 Sep 2015 23:01:32 GMTA few people have noted that when I chair a Fedora meeting, I seem to move quickly and remain strictly focused on the current set topic. I discourage broader discussion on semi-related topics and tend to finish meetings faster than most. There is a reason for this. It is because THAT IS HOW MEETINGS ARE SUPPOSED TO WORK.
Mon, 24 Aug 2015 19:10:31 GMTI started day 2 of LPC with the Graphics, mode setting, and Wayland microconference. This was an extremely dense microconference with lots of information on what is missing in various areas and what has been in the works for some time. To be honest, I felt out of my depth several times. However, it was very clear that the participants in the session were making good progress. One side-note: nobody likes to use mics :).
Fri, 21 Aug 2015 16:00:08 GMTI have the privilege of attending the Linux Plumbers Conference in Seattle this week. This is by far my favorite conference. The talks and microconferences are all of high quality and the events are very well organized.
Mon, 20 Jul 2015 14:48:17 GMTUsers of the Fedora exploded kernel git tree might have noticed that it has been stale for a couple of weeks now. What they might not know is why that is, and when it is going to be fixed. The answer is somewhat complicated and I'll try and summarize here.The Fedora kernel team recently tried shifting to using the exploded tree as the canonical location for Fedora kernel work. The benefits and ideas were written here, and most of those still stand. So I went to work on some scripts that would make this easier to do. The results weren't terrible. Things worked, kernels were still built, and the exploded tree was spit out (albeit at a different location). By some measures, this was a success.Except it really wasn't. Some of the motivation behind the change was to increase participation and transparency around the Fedora kernel. However, while we worked through the new process we quickly realized that it was much more cumbersome to actually produce kernel packages. Since the primary output of our team is packages, that seemed like a pretty bad side effect. Similarly, transparency wasn't decreased but the change made things much more confusing. Changes in the exploded tree weren't synced 1:1 back to the Fedora package repo, so it appeared like big code drops there instead of discrete commits. Telling people to look at the exploded tree was an option, but in Fedora's package-centric world it was a bit out of place.So what do you do when you've spent time making a change and it isn't working out? Well, you scrap it. There's little point in pushing on with something that even the primary author finds to be awkward. The payoffs weren't likely to materialize in any sort of timeframe that would make it worth the continued effort there. Was the time wasted? In my opinion, no. It answered some questions we've had for a while, and showed us that both the tooling and processes Fedora uses aren't really amenable to working with exploded sources.Unfortunately, the exploded tree has been stagnant since we changed back. However, I don't think it will remain that way forever. We tweaked some of the things we do to build the kernel package such that taking that content and creating an exploded tree will be easier. With a bit more scripting, it should be possible to almost automate the creation on every successful build. I'll be working on that off and on for the next few weeks. Hopefully by Flock I have something cobbled together to get things back on track again.It's been a while since I've had what I would consider a pretty big failure. I make mistakes all the time like everyone else, but those are generally small and on short-term things. In a discussion with someone, they asked me if I was bothered by this being a big bigger. I'm not. Personally, I don't care if I fail spectacularly or not as long as I learn something from it. I think I did, so the exercise was worthwhile to me. Failing fast is a really good way to work through some complicated scenarios, and is far better than staying stagnant out of fear of failure. Time to move on to the next idea!(A couple of notes:)It was pointed out on the list that RHEL tooling can build from an exploded tree, but it seems it does this with some odd hacks that we'd likely try and avoid in Fedora. The workflow between RHEL and Fedora kernels are also massively different. I would love to see some more commonality between the two, but not at the expense of forcing ill fitting process on either one.Pagure, is where the exploded tree was hosted in the interim. We tried it with the idea that making multiple committers there would be easier than kernel.org. It is a neat service, but I'm not sure we'd really use many of the features it was designed around. The only complaint I heard was that browsin[...]
Thu, 16 Apr 2015 14:36:11 GMTAs Ryan noted yesterday, Fedora 22 is on track to ship with the 4.0 kernel release. So what does that mean in the grand scheme of things? In short, not much.The major version change wasn't done because of any major feature or change in process or really anything exciting at all. Linus Torvalds changed it because he felt the minor version number was getting a bit large and he liked 4.0 better. It was really a whim more than any thing contained within the kernel itself. The initial merge window builds of this kernel in Fedora were even called 3.20.0-rc0.gitX until the 4.0-rc1 release came out.In fact, this kernel release is one of the more "boring" releases in a while. It has all the normal fixes and improvements and new hardware support one would normally expect, but overall it was a fairly quiet release. So version number aside, this is really more of the same from our beloved kernel development community.However, there is one feature that people (and various media sites) seem to have keyed in on and that is the live patching functionality. This holds the promise of being able to patch your running kernel without rebooting. Indeed, this could be very useful, but it isn't quite there yet. And it also doesn't make a whole lot of sense for Fedora at this time. The Fedora kernels have this functionality disabled, both in Fedora 22 and Rawhide.What was merged for 4.0 is the core functionality that is shared between two live patching projects, kPatch and kGraft. kPatch is being led by a development team in Red Hat whereas kGraft is being developed by a team from SuSE. They both accomplish the same end result, but they do so via a different approach internally. The two teams met at the Linux Plumbers conference last year and worked on some common ground to make it easier to merge into mainline rather than compete with each other. This is absolutely awesome and an example of how new features should be developed upstream. Kudos to all those involved on that front.The in-kernel code can accept patches from both methods, but the process and tools to create those patches are still being worked on in their upstream communities. Neither set are in Fedora itself, and likely won't be for some time as it is still fairly early in the life of these projects. After discussing this a bit with the live patching maintainer, we decided to keep this disabled in the Fedora kernels for now. The kernel-playground COPR does have it enabled for those that want to dig in and generate their own patches and are willing to break things and support themselves.In reality, we might not ever really leverage the live patching functionality in Fedora itself. It is understandable that people want to patch their kernel without rebooting, but the mechanism is mostly targeted at small bugfixes and security patches. You cannot, for example, live patch from version 4.0 to 4.1. Given that the Fedora kernel rebases both from stable kernel (e.g. 3.19.2 to 3.19.3) and major release kernels over the lifetime of a Fedora release, we don't have much opportunity to build the live patches. Our update shipping infrastructure also isn't really geared towards quick, small fixes. Really, the only viable target for this functionality in Fedora is likely on the oldest Fedora release towards the end of its lifecycle and even then it's questionable whether it would be worth the effort. So I won't say that Fedora will never have a live patch enabled kernel, but there is a lot of work and process stuff to be figured out before that ever really becomes an option.So that's the story around the 4.0 kernel. On the one hand, it sounds pretty boring and is likely to disappoint those hoping for some amazing new thing. On the other hand, it's a great example of the upstr[...]
Tue, 27 Jan 2015 15:22:00 GMTAs you might have seen Paul blog about, Red Hat has an immediate opening for a Fedora kernel maintainer position on my team. This is actually a fairly rare thing, as we don't have a lot of churn in our department and most of the engineering positions we hire for are primarily RHEL roles. If you have kernel experience and love working on fast-paced and frequently updated kernels, then this might be a good role for you.
Tue, 20 Jan 2015 14:49:51 GMTTLDR: Use 32-bit x86 kernels? Want to keep using them? Want to make sure they continue to work? Please help!My kids recently convinced me to get them Minecraft. This in turn has caused lots of discussions about Minecraft. In an effort to be able to have anything close to resembling a coherent conversation with them, I've been playing the game myself a little bit. If you've been living under a rock like me, you might not know anything about it. I'll spare you all the never-ending details, but there is one part that recently got me thinking about Fedora and 32-bit and kernels.See, in the game there are zombies. They're not particularly dangerous by themselves. They're slow, and they kind of moan and come after you but you can usually deal with them without really any effort. They only come out at night, and if you catch them outside at sunrise they burst into flames. Unless there's a large group of them, you basically ignore them.In Fedora, there are x86 machines running 32-bit kernels. They're not particularly dangerous by themselves. They're slow, and they kind of stumble around a lot. If you shine a light on the dim corners of the kernel dealing with that code, it usually bursts into flames. Most upstream developers ignore them. Clearly they're a lot like Minecraft zombies, except there are always lots of them and they are never, ever the same.This makes dealing with 32-bit kernels in Fedora actually fairly difficult. With upstream focusing almost entirely on x86_64, there isn't a massive amount of interest in solving 32-bit x86 kernel problems. That isn't to say that huge issues won't be fixed, but they are clearly not a priority (fortunately, they are also rare). There are other cases where the standard advice is to not use 32-bit kernels for things. Have more than 2GiB of DRAM? Don't use 32-bit kernels because PAE is a hack. Want to run VMs? Don't use 32-bit kernels. Transparent hugepages (or hugepages in general)? 32-bit is not your friend.Then there is the variety of workloads people are using 32-bit kernels for. Some of them are old laptops that have crusty ACPI implementations. Some of the are 32-bit servers that are running constantly and stressing various things. Crazily enough, some people even run 32-bit kernels on 64-bit CPUs. That last one is a pet peeve of mine, but I won't dive into that here. The ISA variety is a headache as well, with some CPUs not supporting PAE so that we have to build different kernels for i686 and PAE-capable i686 machines.When you take the above, add in the bug backlog we get from the just as widely varied x86_64 machines, the fact that our 32-bit hardware is rather limited for testing, the 32-bit x86 kernels in Fedora are simply pretty low on the priority list. We build them, we make sure we grab any fixes we see or are pointed to for them, but in the larger picture the time we spend on 32-bit specific issues isn't benefiting the majority of Fedora users. So the kernels linger on.Not surprisingly, I'm not the first person to notice this. Just today I've had 2 discussions on what to do about i686 in Fedora, and Smooge posted his idea for a way forward. Others have had similar ideas. RHEL 7 does not include 32-bit kernels at all. I'm not going to comment on those proposals yet, but it does seem to at least confirm a bit of what we see on the kernel side of things.So what can be done here? Should we kill the 32-bit x86 kernels? Should we kill one of them? Do we spend time on a solution I previously thought about a long time ago? I don't have answers for all of that at the moment. However, in listening to a very detailed dissertation on Minecraft zombie solutions from my son, his last solution w[...]
Thu, 09 Oct 2014 14:45:10 GMTAnyone that has been around Fedora long enough knows that FESCo is the technical steering committee. As Fedora committees go, I think it's one of the best. It works well, it remains on task, and it has typically been a good example of how to get things done in Fedora. But it's not infallible. Like everyone, sometimes even FESCo can be blinded by its own purpose. Yesterday was, in my opinion, one of those times.
Wed, 01 Oct 2014 22:48:50 GMTAs mentioned in my last post, a number of people over the years have asked about an exploded source git tree for the Fedora kernel. We've never done this, primarily because our build tool wouldn't be able to use it, and it didn't really have any perceived value for the maintainers. However, it's come up enough times that I thought I'd give it a shot. Creating it was easy enough, but whether it winds up being useful to anyone is something that you'll need to help with. This post will cover what it is, how it works, where it is, and possible future plans.For the TLDR, see the bottom.What is it: This is maybe the git tree you are looking forIf you've ever worked on packaging software with RPM (or apt or a number of other package managers), there are two basic ways to do it. You can use the upstream release tarballs and apply patches via a spec file, or you can use a snapshot of the upstream development repository as the "tarball". The former is how the Fedora kernel is packaged, and it presents a nice separation of upstream and the patches Fedora has added. However, that means that every time a new upstream kernel is released we wind up rebasing our patch set on top of it. When doing the initial creation of the git tree, I thought about a number of ways to handle this.We could use feature branches for each patch/patchset and do a merge in the main branch. That would be how upstream development works for patchsets, but some of our patches are rather long-lived for various reasons. Doing merges like this over time would lead to a very tangled history and it wouldn't really provide much benefit to anyway.Another way would be to start with a base version, add our patches, and then merge in the upstream kernel changes on top of that. This is essentially the opposite of the scenario above. It still results in a rather tangled history over time, and it actually sends the wrong message: this git tree isn't an upstream. It's supposed to be a downstream representation of what we've added/changed. We don't want people developing new kernel drivers/features against the Fedora git tree. We want them to develop against upstream.Another factor in figuring this all out was our build system. As I mentioned, koji isn't setup to take random exploded source git trees as build input. It expects things from particular places and in particular ways, which is perfectly fine. To make this git tree a "source" for koji, we'd have to use it as a snapshot tree and not list out the patches and such in the spec file. There are distros that do this, and maybe that's suitable for them but I find it valuable to look at the SRPMs from a distro and be able to easily see what's changed [sideways looking emoji].So in the end, that means the git tree became an exact mirror of what happens to the kernel RPM in Fedora. Namely, it rebases on top of whatever upstream is doing. Now typically this is a terrible thing to do to downstream consumers. It means they need to force update their tracking branches and such. I'd be very hesitant about it if not for the fact that it's mostly just an ease of use thing. Another point is that it's analogous to how linux-next operates (albeit for different reasons). Overall, I think this is the best trade off between "git tree exists" and "maintainer remains sane". The follow section should help illustrate why as well.How it works: all your rebase are belong to usHow it works is pretty simple. The tree tracks the main upstream git repo in the master branch, and the Fedora content for a release is contained on the "rawhide" branch. As of right now, only the rawhide kernels are tracked here, mostly due to[...]
Sun, 10 Aug 2014 18:23:52 GMTIt is traditional when you go to a Fedora conference to do blog posts about your experience there. I think they're great, and I've enjoyed reading those posted by others. There are some great talk recaps and it's always interesting to see how people are spending their time. For one reason or another, I'm not one of those that manages to do this on a daily basis though. That usually means I try and do a week long recap of what happened, and the information is largely stale or redundant and I don't think it makes for great reading or communicating. I thought about this a bit and I think instead I'm going to talk about what I'm planning on doing now that Flock is over that came as a direct result of things that occurred while I was there. Hopefully that will be a bit more entertaining and really keep me honest with those plans.Custom Kernel buildsA number of people came up to me and asked how to build custom kernels. The instructions we have on our wiki page are probably fairly valid, but I really expect that they are stale in certain areas that lead people to have trouble. The kernel maintainers never really build kernels the way the wiki page suggests, so it might be time for a revamp. I'll look this over and give some examples of common cases (enabling config options, adding patches) and hopefully this will lead to people having fewer issues.A Fedora kernel git treeWe've often talked about having an exploded source tree. The problem with that is that it's of little benefit to the Fedora kernel maintainers. Koji builds from the SRPM and that's from the tarball, patches, and spec file. However, getting a kernel tree created is likely only difficult from the start, and keeping it maintained could probably be automated. It would have to rebase the branches, as that is literally what we do in the SRPM, but if people find it useful and it isn't overly difficult to create, we might as well try. Plus, it's a good way to spend a day or two when I need a break from the daily grind.Packaging changesA couple people spoke with me about some possible packaging changes for the kernel. Things like the kernel-source package that Fedora shipped long ago. To be honest, not all of these will probably happen but there are a few minor things we'll likely change.Bug workflowUnsurprisingly, this is one of the larger topics I left with things to work on. For starters, we want to try and automate some of the triage we do. It shouldn't be massively difficult to write something that parses an oops and figures out the correct part of the kernel it came from. We'll likely start with ABRT bugs, since those are mostly formatted in a uniform way and work from there (we might even be able to get ABRT to do this at some point as well.) We can also pull from the retrace server once that is able to filter out results. This is a pretty broad description of what to do, and that's because there are many options and directions we can take. Hopefully some automation will really help us get time back to work on different things.Rawhide kernel handlingIn my kernel talk, Owen asked me "Is the rawhide kernel the right kernel for rawhide?" He was referring to the fact that we build with debug options enabled, and this has performance impacts. Those impacts are less noticeable to a human now that we leave SLUB debugging off, but if you're using perf or some automated performance metrics it is going to be easy to see it as slower than a normal build. Even ignoring the performance aspect, there are still days where we'd like to get a build out for "adventurer" testing before it hits the main rawhide compose, but we d[...]
Tue, 08 Jul 2014 19:25:34 GMTA while ago, we had a thread on the Fedora kernel list where people were expressing a desire for easily accessible kernels that contained new and shiny technologies still under development. We've had a number of discussions like that over the years, and none of them ever seem to pan out to something remotely feasible. The patches in question tend to be larger, with no clear upstream path. The one thing we've learned from uprobes, execshield, and now Secure Boot is that carrying non-upstream patches is kind of a nightmare. It makes us different from upstream, creates a maintenance burden, and actually leads to those patches taking even longer to get upstream because there's no impetus once they're "in-distro". So we've been sufficiently hit with the cluebat enough to say "no major non-upstream patches" and to be honest that approach has been working well.Except that precludes people from working on them, getting userspace ready for them, and improving them. There's a catch-22 there. The features aren't making upstream progress because nobody is using them, but nobody is using them because they aren't easily accessible. Not everyone is willing to build their own kernel just to play with a new toy. Fedora hasn't been willing to take the hit so they can easily use something that might not actually get upstream (see aufs). So what can anyone do?The Fedora kernel team has thought about this for a while. How do we deliver something people are asking for without impacting the rest of the distro. The rawhide nodebug repo has shown that simply delivering a side-band kernel doesn't drastically increase the maintenance burden, but that isn't really a great comparison. That is literally the same kernel, just with the debug options disabled. However, the introduction of Coprs in Fedora gives us another option.The kernel team first used a Copr to work on the kernel-core/kernel-modules packaging split before landing it in rawhide. The Cloud team was able to point their instances at that repo and test it out. This actually worked out really well and we fixed a number of bugs before they hit rawhide. The content of the kernel was still a 100% match to the mainline Fedora kernel, but the packaging difference was enough to prove the mechansim worked well enough.So with that in mind, I've decided to create a kernel-playground Copr. The idea behind this is to provide easily consumable kernel builds with some of the things people have been requesting for a while now. Before we get to specifics, here's the playground rules: * This is not a "supported" kernel. Bugs filed against the Fedora kernel component with the kernel-playground kernel will be closed. If the bug clearly lands in one of the features being carried, we'll make attempts to forward the information to the relevant upstream developers. If there are issues we can still discuss them, but we want to keep the Fedora distro and it's process/tracking isolated from this like we would with any other non-standard kernel. * The kernel-playground build will roughly track rawhide, with selected features/patches added on top. I say roughly because it will not receive a build for every rawhide build done like the nodebug repo does. The tentative plan is to update it once per upstream -rc and final release.* x86_64 only. In the future, other 64-bit architectures might be possible. * There are no guarantees around feature/patch additions. We'll do our best to keep things integrated, but we might have to drop additions if they become too much work. There are also no plans to "graduate" features from the playground k[...]
Tue, 20 May 2014 14:42:44 GMTA brief release overview of the past several weeks below. If you have any questions, please chime in!
Mon, 17 Feb 2014 18:52:51 GMTI recently had the privilege of attending DevConf 2014, and it was really impressive. This was the first time I had ever traveled to the Czech Republic and I found the people to be extremely pleasant and helpful. The city of Brno was really easy to get around and the conference accommodations were nice.The conference itself was more than I expected it to be, both in terms of scale and quality. That's not to say that I expected it to be poor quality, but at all conferences there are some talks or tracks that wind up being less worthwhile than you expect. That wasn't the case here. Every talk I attended was well done and very informative. Despite the conference organizers warning that the presenters were engineers and not public speakers, I thought everyone did a great job. Clearly this is a continuing trend because the conference was packed with people. It was almost to the point where a larger venue would be needed.Day One The first talk I attended was Thomas Graf's Linux networking stack overview. He took a very complex topic and presented it in a clear manner without overwhelming people. It was very well done, and I'll probably refer back to his slides when looking at networking issues for quite a while. I really like these kind of overview talks, because while I stare at kernel code all day long, it's hard to get a bigger picture of the whole stack if you don't work on it daily. Very helpful.After that I attended Alex Larsson's talk on Docker. Docker was everywhere at the conference and the hype around it was pretty evident. Being a kernel guy not very much in tune with the higher level userspace stacks, I wanted to understand why people were excited and what usecases might be present for Fedora. Alex did a great job of talking to that hype and then was brave enough to give a good demo that worked. His talk was great and while I might not see myself using Docker in the future, it was definitely informative enough for me to explain Docker to someone else later on.It was interesting to go from the Docker talk to Colin Walters' talk on OSTree. Where Docker seems designed to make it easy to swap out "stacks" of things, OSTree is very focused on building an immutable tree and having that be the basis for testing and deployment. The ability to build the trees on the server and deploy them on both bare-metal and virtual machines is interesting to me. I can certainly see this being very helpful as Fedora is making progress towards a release. It removes the ambiguity around what "version" of Fedora you're testing, since the trees are named and immutable whereas a typical system on Branched is a collection of packages that may or may not match the latest releases. I think this could also provide a great base for things like cloud-images, where you don't really focus on running an install for a great duration and upgrading it as you go. I really appreciated Colin's approach to the presentation as well. He presented both benefits and downsides to his ideas, which is not something you often see from someone presenting a new technology.After that I did a bit of "hallway tracking". Catching up with people I haven't seen in a while, and meeting several that I've never met in person for the first time. As is typical with most conferences, the hallway track is very useful and this was no exception.I went to the Continuous Integration with OBS talk, given by Adrian Schröter from SUSE. It was a very interesting presentation. Being able to see what OBS does differently and in-addition to koji is always[...]
Tue, 14 Jan 2014 19:30:06 GMTThis post isn't about my kids, but they're the prompt for it so bear with me.My kids participate in an after school program that is designed to foster creative thinking. They do spontaneous problem solving and "outside-the-box" kind of thinking. Watching them do this and hearing about the ideas they come up with has been pretty interesting so far. Their solutions to problems aren't always practical, but it's their interaction with their team members that I really noticed.They are the youngest two kids in the group, by at least 2 years and in some cases 4. Sometimes when the coach gives a problem, the older kids say "oh, we can't do that because " or "that won't work because " etc. The coach is great and works really hard to get them to think harder, but she said my kids very rarely do that to begin with. Instead they just throw out all kinds of ideas. Need to simulate water? Use lots of blue M&Ms. Need to make something look smelly without actually making it stink? Use pipe cleaners to create comic style scent waves coming off of your body. Some just aren't feasible to do, but then the group discusses the ideas and figures out _why_ they aren't really practical. And a lot of the time the ideas are great. All it took was an unjaded (and sometimes naive) point of view.That got me thinking about Fedora and open source in general. Fedora is 10 years old now, and despite the Fedora.next stuff we're currently working through, we're pretty entrenched in our ways. We have RPM and packaging guidelines and processes. They've worked to produce 20 versions of Fedora, and there is a lot of momentum behind them. However, that doesn't mean they're the best long term thing going forward.We have developers writing applications that bundle things. We have people wanting to make app containers that are portable so they're distro-agnostic. We have things flying in the face of "Unix tradition". The response to a lot of this stuff within Fedora (and other distros) has been mixed, but there is quite a bit of "we can't do that", or "it's too hard", or . Even the Fedora.next stuff is met with a lot of resistance based on the grounds that it's a rather large and significant undertaking. There are lots of people saying "it can't be done" or that it won't matter. They may be right. They may also be wrong.Even within the kernel team, I tend to have knee-jerk "no" answers. Usually with good intentions, but often without really exploring it in detail. It doesn't really help expand my knowledge base and it probably leaves proposers of things with a sour taste in their mouths. I don't know anyone that likes being told a blanket "no" without an explanation.Thinking about all of this made me wonder if we're missing some great ideas because we've lost that unjaded point of view. So I think from now on I'm going to try pretty hard to consider suggestions with a more open mind. If I think something isn't really practical or feasible, I'll try and respond with reasoning as to why or I'll respond with what it would cost to do and let the proposer work that out. Maybe this won't really lead to anything new, or maybe it will lead to doing something new that ultimately fails. But at least we have the possibility of making something better or learning from a mistake. I guess I'd rather have that than sit around being stagnant and eventually irrelevant. Here's to thinking outside the box.[...]
Mon, 16 Dec 2013 19:46:20 GMTLately I've seen a lot of weird references to the kernel come up in Fedora discussions. These range from basing things on its upstream release cycle, to somehow holding it up as an example of some strange patch/development model. It's really kind of baffling and also kind of frustrating. So here's the secret about the Fedora kernel: it isn't any more special or different than any other package in Fedora. It's a package.The Fedora policy for kernel patches is "upstream first, backport when relevant", like most of the rest of Fedora. There is no development done directly on the Fedora kernel at all. Patches that are included in the kernel package are already mostly backports, and the few that aren't are either dumb things to silence warnings or change defaults. There are exceptions here and there for major features, but those are almost always posted by their authors upstream at the same time (and the authors are rarely the Fedora kernel maintainers). We don't want to carry major out-of-tree features for any longer than we have to for all the same reasons you don't want to carry a ton of patches against any other upstream release. It winds up benefiting far fewer people to do so, and increases the maintenance burden on Fedora.The upstream release cycle is fairly regular, which is a good thing. However, that is done to benefit upstream, not anyone else. It's also rather short, so basing something else on when the upstream kernel releases would likely introduce a lot of significant churn for really no reason. If you're developing a Feature for a Fedora release that requires kernel changes, we're going to tell you to get them upstream. If you do that with sufficient planning, then we're likely to get that upstream release in the Fedora release anyway. If for some reason it doesn't line up, then we can look at carrying it for a while. But to base some other release schedule on the upstream kernel release is really kind of odd. Just don't do it. Plan for whatever thing you're trying to release by figuring out what problems/features you're trying to solve, how long it's going to take, and how much testing you need.The Fedora kernel also has a lot of bugs in bugzilla. Maybe more than most packages, but what _isn't_ different is that we're working as best we can to resolve them. Holding up the Fedora kernel as a do or don't example of bug handling is really no more relevant than any other package. Some bugs we don't get to. Some we have no clue on. Some we fix. Some we fix and then there's a regression. All of these things happen (or can happen) in every other package in Fedora. We aren't doing anything magical here. (If you have bugzilla magic, PLEASE LET ME KNOW.)The kernel can cause your machine to not boot. This is true! However so can glibc, dracut, shim, grub2, grubby, systemd, lvm, dbus, and a number of other important packages. Like all of those, we strive to avoid breaking things so people can use their computers as they'd like. Most of the time we're successful. Sometimes we aren't. In the times we aren't, it's normally not a Fedora specific issue and instead is something we need to fix upstream. Exactly like the other packages."But the kernel gets major version updates in the middle of a release", you say. It does. This is, admittedly, slightly different than how most packages work in Fedora. We've discussed this several times elsewhere, but really the best way to think about it is that it's done because of scale. Each[...]
Fri, 16 Aug 2013 13:29:04 GMTFlock was a great and exhausting conference. I can honestly say, it is one of the best Fedora conferences I've been to. Now, being one of the people helping out with it (in a small fashion), that might sound biased but I've had many other people tell me the same thing. I'm really glad people found it worthwhile, fun, and productive. It makes the work put into it much more rewarding.
Fri, 17 May 2013 17:28:41 GMTThis week we took a patch from Harald Hoyer to the Rawhide kernel.spec to switch to using the kernel-install tool to install the kernel onto the system. This new tool is provided by systemd and is intended to be distribution agnostic, with the eventual goal of getting into the upstream kernel Makefile as well.
Thu, 09 May 2013 18:29:09 GMTEvery once in a while, I crawl out from under the rock that is bugzilla and I try and look around at what others are doing in the distro kernel space. Today I was curious how Fedora and Ubuntu compare in how they configure the kernel. I've long thought that for all the focus the kernel gets, it should be the most boring package in an entire distro. It should work, work well, and that is about it. It isn't there to differentiate your distro. It's there to let your distro run. So, will my personal belief stand up, or would I find something in the configs that proves one "distro" is better than another? Let's dive in.I won't give a line by line comparison because 1) that would be very long and 2) that would be utterly boring. You don't need to know that e.g. Ubuntu enables one scsi driver that Fedora doesn't or vice versa. Instead, I'll try to focus on various functional areas and give an overview. The kernels For comparison purposes, I used the config from linux-image-3.8.0-19-generic Ubuntu 13.04 package and the 3.8.11-200.fc18 kernel from Fedora. These are the same base upstream kernel version, and both of them are from the latest stable release of their respective distros. I used the x86_64 (amd64 in Ubuntu speak) build, as I found that to be the most relevant in this day and age. I could have used the kernels under development, however those are more apt to change and I thought it would be better to compare things out there in wider use. Overview The vast majority of the config is pretty much identical. It's what you would expect from a distribution config. There is a wide range of support for all kinds of devices and buses, with much of it being provided by modules instead of built in. This should be expected, given that they're coming from the same upstream code base and really there is only one sane option for most config options. Low-level options A few of the low-level options did differ though. One of the more notable ones is that Ubuntu has CONFIG_NR_CPUS set to 256, whereas Fedora has it set to 128. On the face of it, one might think both values are overkill as people aren't booting on machines with 128 cores very often. However, Ubuntu has transitioned to having a single kernel between it's desktop and server variants (starting with 12.04 I think), so it does make some sense for them to have it set higher. Fedora can revisit our choice if we get a sudden clamoring for massive machine support. I await my mini-supercomputer test system.Another area where we differed was in the NUMA options. Ubuntu enables CONFIG_X86_NUMACHIP, CONFIG_MOVABLE_NODE, CONFIG_MEMORY_HOTPLUG, and CONFIG_MEMORY_HOT_REMOVE. The CONFIG_NODES_SHIFT value was higher on Ubuntu as well, allowing for more NUMA nodes. Again, given their server variant that probably makes sense for them. Fedora just doesn't deal with that class of machine on a regular basis.The HZ area also had some differences. Fedora defaults to HZ_1000 compared to Ubuntu's HZ_250. That means the timer tick is 1000 times per second on Fedora, versus 250 on Ubuntu. Typically a higher HZ value is used for better interactive response. On a server system where there isn't typically a desktop or user waiting for interaction, a smaller HZ setting allows more work to get done as there are few timer interrupts to process. Of course, both distros set CONFIG_NO_HZ, which means the timer interrupts are only triggere[...]