Subscribe: Crashing Into Open Source (without a paddle)
Added By: Feedage Forager Feedage Grade A rated
Language: English
bug  bugs  bugzilla  make  mozilla  new  open  people  push  results  syntax  things  time  web  women  work  working 
Rate this Feed
Rate this feedRate this feedRate this feedRate this feedRate this feed
Rate this feed 1 starRate this feed 2 starRate this feed 3 starRate this feed 4 starRate this feed 5 star

Comments (0)

Feed Details and Statistics Feed Statistics
Preview: Crashing Into Open Source (without a paddle)

Crashing Into Open Source (without a paddle)

Where I share my adventures as a Mozilla Build & Release Engineer and keep notes on my interests, participation, and development in F/LOSS.

Updated: 2018-04-15T03:10:53.941-07:00


Moving my virtual home


Head on over to for all future writing. This blog will stay up as long as Blogger keeps the lights on, but everything here has been transported over to the new site, so don't worry - I have backups!

Autolanding your patch(es) to Try via Bugzilla


We're ready for a soft release of the first step in our very experimental autolanding system. Experimental meaning: we reserve the right to pull the plug, take it down for tweaks, and some information may be lost when bugs arise.  You can check the if the autoland system is up and running by going to

This is the Try branch only portion of what will become a system to automate and more easily manage multiple branch landings.  Marc Jessome, our returning Release Engineering intern, and myself have this as a Q1 goal.  There are several issues to be resolved with this system and the link above will keep you appraised of what the goals and known issues are. We will be working hard over the next two months to make this a secure, reliable system for landing patches with minimal developer time spent pulling, pushing, and watching tbpl.  The hope is that it will also become a useful tool for Release Coordinators to track that a fix lands across several branches as needed.  Future features will include a bugzilla extension to securely interact with this system and remove the need for whiteboard tag changes and a RESTful API so the whole process could be initiated through the command line.  We appreciate constructive feedback but please hold off on scope creep suggestions :) 

Here's how it works right now:
  1. Attach patches to your bug Note: if you (the attacher) do not have permission to push to Try, get an r+ from someone who does
  2. Use this documentation to determine the appropriate whiteboard tag and enter it in the bug so our autolanding poller will pick up on your request
  3. The autoland system, which tracks all bugs with autolanding requests, will queue up your patchset and then pull it from the queue, apply the list of patches in your bug (or those in the whiteboard tag) to a clean pull of mozilla-central tip, commit each patch as the patch author, and finally push to try as Autoland User
  4. A comment will be posted in your bug with the results (hopefully successful) of the push attempt, and the whiteboard tag will read [autoland-in-queue]
  5. After all builds complete, their results get posted back to your bug (just like with the current '--post-to-bugzilla Bug #' flag in trychooser) and the [autoland-in-queue] whiteboard tag will be removed
There is a threshold on how many bugs the autoland system will push and track at a given time.  We are curious to learn about usage and load on this system so please give it a try and inform us of any issues in IRC: #build or #developers channels are the best places to find myself (lsblakk) or Marc (mjessome) and chat us up about this initiative.

Growing the company, structuring volunteerism: My response to David Eave's community lifecyle audit


Last Wednesday David Eaves presented the results of the multi-tiered contributor lifecycle audit (watch the video).  A few points really grabbed my attention and as someone with a background in arts & education non-profits I feel the need to share my experiences alongside my reactions to this talk. David pointed out that as we are growing, we can hire people to solve problems, so what exactly do we need volunteers for? Survey results showed that contributors often don't feel their contributions had much impact on the project and that as our paid staff pool grows in size, there is less clarity about what exactly a volunteer's importance is to the critical path of Mozilla's mission.  I wish we had this data prior to the 1+ year push to releasing Firefox 4.  My hand-waving hypothesis would be that as we were cramming to get a product out the door we forgot to be leaders of volunteers and instead unconsciously pushed them aside so that things could get done on a schedule.  It was a stressful time, but now that we've moved on to regular, rapid releases perhaps we paid staff can all take a step back and really assess how we incorporate the work of volunteers into our individual areas.For many Augusts I have worked at a women's music festival in the woods of Michigan and at that festival there is a kitchen that has pumped out 3 vegetarian meals a day for 5 days to 4000-8000 attendees depending on the year.  This festival relies on a core group of 'workers' who are in fact volunteers but let's pretend that group of 500-600 people is like the paid staff of Mozilla.  The main kitchen work crew gets about 30 workers to run the kitchen.  You might think to yourself "but 30 people cannot produce enough food for 4000-8000 women" and you'd be very right.  The way it works is that all attendees of the festival are asked to do two 4 hour workshifts during the week they attend the festival.  The majority of workshifts center around the main kitchen and creating the meals which range from burrito night to pasta putanesca to nutloaf (a vegetarian version of meatloaf). All these meals involve preparation of pounds upon pounds of vegetables as well as cooking of pasta, beans, sauce.  Did I mention this was all in the woods, over a massive firepit?  Yup, so 30 women are in charge of making that entire process work and they do so by wrangling hundreds of volunteers per day into shifts around each meal and constantly leading and dividing up the work so it can be done by many hands yet results in one huge meal - 3 times a day!So let's go back to people who are volunteering not feeling clear about how Mozilla works and whether their time and effort has impact.  How can we make sure that the smallest task makes that person feel like they've contributed?  Some areas of Mozilla do this very well to name a few; AMO, SUMO, QA, and Personas. These teams manage tons of volunteers and have tasks with measurable outcomes (tests run, filing bugs, approving an add-on, answering a question in the knowledge base, shrinking the queue of pending Persona submissions) and sometimes they can use scoreboards or themed days to get volunteers mildly competitive for the respect of their peers and accomplish larger goals in a short burst. I'd encourage people interested in having volunteers participate in their team's work think about how to make sure their volunteers have tools to measure their impact from the get-go.  In Release Engineering I would measure the number of bugs that we have yet to fix, many of them labelled "simple" or "old".  If I had volunteers doing RelEng work I could keep track of their progress and post reports (as blog posts) of who fixed what when and how as the number of bugs shrank.Another interesting result from the survey: older folks (34-55!) aren't on-ramping as much as younger ones. At first I wondered how much of this was about access to the muscle memory of being a student.  I think it's fair to assum[...]

OccupediA - Women Contributing to Wikipedia (the first of many such events)


Last Thursday night about 8 women arrived at Noisebridge to learn how to contribute to Wikipedia.  Several things led to this gathering:An article in the New York Times back in October drew attention to the lack of women contributors to the Wikipedia knowledge base and that got me thinking.  Having organized other spontaneous "women get together and learn stuff" events I figured I could take the same approach to Wikipedia contributing, get some women together to create accounts, generate content, learn how to stop vandalism and see what would stick.  Recent participation in activism around the Occupy Wall Street movement also inspired me to try and reach out to communities I am in who are not as technical, to encourage people to come first with knowledge and interest in topics Wikipedia could benefit from and let the tech come second. A month ago Elsa and I were talking casually about all the the above mentioned things and we decided to just go for it and pick a date, throw it up on the Noisebridge (local SF hackerspace) calendar, and see what we could make happen.We took over a small makeshift classroom space at the back of Noisebridge. It had one lamp as the primary source of light because the fluorescent holders above were missing their tubes.  A man was near the back working on a dress for fashion school, several other hackers were up front working on their various projects.  Noisebridge was a wonderful place to have this event. It feels like anything is possible in a space like that.I was happy with the turn out - we had a mix of artists, educators, and tech workers. Also as a bonus one of the attendees, my coworker Boriss, was a seasoned Wikipedia contributor who was able to really detail the ins and outs of the different levels of participation.  I can't stress enough how amazing it was to have her and her knowledge there because there are lots of misconceptions about Wikipedia (I definitely had some) and her first-hand knowledge was inspiring to me.So the beginning of the meetup went well enough, and as you might expect.  We introduced ourselves, talked about why we had come to the event and what we were hoping to get out of it. We started in on learning how to set up an account if one didn't already exist and we looked at discussion/history/edit and other basic navigations of Wikipedia space.  There were a lot of questions about what belongs in Wikipedia, neutral tone, citations.  The conversations were lively and I found them quite enjoyable.Here's what I didn't expect: Getting folks interested and excited about Wikipedia becomes REALLY HARD in practice.  Unlike learning Python where the participants can hammer out some code on their own computers in minutes and feel accomplished, there is a lot more complexity to Wikipedia.  There is a lot of confusion about their UI, their purpose, who can do what and when. Very quickly it seemed that the women who had come to the event feared adding anything new to the knowledge base and they were also incredibly intimidated by the UI of the site. It wasn't even clear enough how one would create a new article when none existed.From this event I learned a lot about organizing and about the intentions of future events like this and I did a little braindumping while we were meeting so I could remember to list them later in this very post.Things that would help newcomers:Having a "new to wikipedia" moniker next to their nickname for the first N activities on the site (we have this on our Mozilla bugzilla) so that hopefully older and wiser participants would be extra nice to themFind a way to make some of the simpler tasks that help Wikipedia (typos, reverting vandalism, categorizing articles) into a game that a new arrival could play that would start easy and then move more toward the real-life workflow of working on Wikipedia - as a way to warm them to the UIEncourage newcomer to write a straight-up article and have a place for [...]

Want more women in Open Source? Donate to Ada Initiative today!


Short version: If you love women, or even like them just a bit, go right now and donate to the Ada Initiative to show the women in your life that you value their contributions past, present, and future to the wonderful world of Open Source. I'm going to make a donation in my grandmother's name this year and I know she'll be happy to have supported such a valuable project.Long VersionI love Open Source.When it first came to my attention, in the first year of my degree in software development at Seneca College, I knew we'd be a good fit.  There's something about the spirit of Open Source that instantly clicked with my existing guerilla activist sensibilities.  The way that you just take what you want and make it happen.  That you create and give away. That you work with other passionate people to make cracks in the surfaces of monopolies that only want you to be able to do things through their (usually financially) gated communities.  It reminded me of how I had approached being a filmmaker - taking $50 of Super 8 film and developing it myself in 16L bucket in a dark bathroom then submitting the results to a prestigious film festival and being accepted.  Having my work shown alongside films with budgets bigger than the cost of a house was an amazing experience and taught me that not everything has to be polished to be valued.Open Source is like that to me, the diamond in the rough.While I was working on my degree I of course noticed (and was not surprised by) the lack of women in my classes.  I was surprised when I started to get involved in Open Source to discover that there were less women in FOSS than in proprietary software companies. That seriously BLEW MY MIND.  I mean, if Unlocking the Clubhouse is to believed (and it is very thorough research) then technical women want to do work that is meaningful and helps people.  Why that sounds a lot like Open Source doesn't it?  So why aren't there more women in Open Source?  I'll let you Google that question to your hearts content, there's a lot written on the subject and so much more could be.  The point though is that the Ada Initiative is a new project that is here to take on that very question through ACTION.  They will DO things to get more women in Open Source.  Women don't have to be dragged into FOSS kicking and screaming.  Trust me, after seeing the overflowing wait list for the Grace Hopper Celebration of Women in Computing's FOSS day, there are a ton of talented and smart women interested and able to do work in Open Source.  We (all of us who have already drank the Kool-Aid) need to help them get integrated and feel comfortable staying in FOSS. When I first met my future team at Mozilla in April of 2008 there was a woman on my team (!) and she self-identified herself to me as a feminist within the first 5 minutes we were together.  As someone who was coming in as a student with zero experience in professional tech workplaces I was so thrilled to have an immediate feeling of relief, trusting that if she was respected there I would be too.  She also introduced me to wonderful internet properties such as GeekFeminism and Sociological Images both of which helped me start connecting with other feminists in tech fields.  Almost three years later I am starting to feel like I've been successful in building the community in FOSS around me that I want to be a part of.  It's a wonderful mix of the talented people I work with at Mozilla, the folks I'm working on planning the next Dare 2B Digital with, the programmers I organize PyStar workshops with, the Women Who Code, the Women 2.0, and of course - The Ada Initiative.I'll leave you with their own words about why you should go straight to the donation page: We’re proud of what we’ve accomplished already.  Since our founding inearly 2011, we helped over 30 conferences and organizations adopt ananti-harassment p[...]

My First Startup Weekend: Women 2.0 Startup Weekend


On November 18th, 2011 I jumped into the deep end of the Bay Area startup culture I have been lurking on the periphery of for the past two years of living here. After going to my first Geek Girl Dinner at Microsoft a month ago, and preparing to talk about women in open source at the Grace Hopper Celebration of Women in Computing, it seemed very much up my alley to sign up for the Women 2.0 Startup Weekend held in SF at The Hatchery.  Originally Angie had asked me to be one of the mentors which, while incredibly flattering, seemed beyond my current skill set.  I do always have ideas for new projects/apps though and I've been trying to get even more full on development experience under my belt so it seemed like a deal to get to spend 54 hours working on a startup idea for $50.[tangent] I love the immersion-as-classroom experience, btw.  I made my first Super 8 film in 1999 at G.I.F.T.S under similar conditions where I lived with my other new-to-filmmaking cohort in a couple of trailers-turned-bunkhouses over on the beautiful island of Galiano and for one week we did nothing but eat, sleep, and breathe guerrilla filmmaking.  We shot, hand-developed, transferred, and then edited our work, cranking out an entire short film in just one week.  I left that experience filled with confidence that I could make a movie a week for the rest of my life!* What I was hoping for out of my first Startup Weekend was an up-to-your-armpits code-a-thon and what I got was...very much NOT that. Here's what I got out of Women 2.0 Startup Weekend instead:Pitching 101Some people had come prepared.  They had read an email I missed, knew what was supposed to be in a pitch, perhaps even had some code or a site name or some idea of what they would need to take the next step into their imagined company.  I had none of these things.  I had 2 ideas, one of which had occurred to me the week before on a bike ride.  I jotted down my ideas quickly and 'pitched' them to a couple of women I knew from other local events (like my CodeChix pal, Vicki).  Both of my ideas seemed to get people interested and with the help of a few very kind listeners, I chose which one I would officially pitch and worked on naming the imaginary app as well as figuring out what salient points I wanted to get across.  It seemed wise to me to get into the early round of pitchers, little did I know that there would be about 67 pitches.  I was #6 and so it took a long time to get to the point of being able to move about the room chatting people up, which I am sometimes not so good at What I did: I pitched it, was too shy to really reach out to strangers and try to woo them to my idea, I hid my sign for a while only taking it out when people asked me about it, I got 7 sticky-note 'votes' for it (which was amazing to me), but I already knew that I would not be working on this project over the weekend and I was shopping around for a team that I would be excited to spend my next 54 hours with.What I will do next time: Work more on my pitch ahead of time, have a clear idea of WHO I would like to join me, go around the room and find people who match those roles, have more research about my 'market' ready to help with the business side of things.What I wish Startup Weekend organizers could improve: Help people match up by roles - so have all the designers go to one corner, all the marketing folks, all the developers, etc.  Give us a visual of who's there looking to do what so that we can more easily go around and network.  It seems less efficient to me that I would have to go chat up 10 people and perhaps discover that none of them are a match for what I'd be seeking. Even putting this info on people nametags would help - especially for folks who have multiple skills they want to highlight.A Team is FormedThe eliminations were happening and I already knew I was going to put my idea aside for [...]

New to IRC? Never tried it?


Do you shy away from IRC because it seems daunting, complicated, or completely foreign to you? A pretty large chunk of the Mozilla online community lives and breathes in IRC so I encourage you to come give it a(nother) try. I recently updated to help get you started. Hit me up with any questions here or find me as lsblakk in IRC :)

I'll re-visit the instructions soon to write about screen & irssi to keep a perpetual connection going in an attempt to make that option more accessible to folks who might feel it's too technical.

Where Are My $project-branch Nightly Builds?


Did you know that we don't build a fresh nightly on a branch unless there's fresh code?  Well, now you do!  In the interest of saving even _more_ resources and network bandwidth so that we can accommodate even  _more_ project branches we have added this little bit of logic to our nightly build scheduler.  It makes sense, right?  I mean, why have a nightly that is just like last night's nightly?

In other news, Ben just added 4 more twigs (aka Disposable Project Branches) for side project work and one of them is still available for booking.

Want to help? Encouraging community contributions to Release Engineering


In a timely confluence with Mozilla's new Steward initiative, I'm preparing to get some community contributors engaged with some of the projects we work on in Release Engineering.  A fair amount of our production infrastructure has to be locked behind VPN and sekrit passwords (we have 400+ million users to protect) but there are more and more RelEng side projects. We provide tools to the larger developer community and solve interesting scalability challenges with our unique (and massive) automation systems that can be worked on by any interested person in their own local test environment and then integrated into our /build repos. My personal goal is to try and get 2 or 3 regular community contributors to come work with us on tackling these.In order to solicit contributions I have been working with David Boswell. We added Release Engineering to the 'areas of interest' page and I have created the beginnings of a RelEng-specific contribution page. The first two areas that I think would be a great introduction to working with RelEng code & tools are the TryChooser and our upcoming Autoland system.  For the latter, our intern Marc Jessome is sticking around this fall as a contributor to carry on the amazing work he put into this system over the summer.  He'll be continuing to debug the code and improve the portability of it so that we can get it into a beta testing stage by the end of October.  As that work is being done we also need someone to help us write the API functionality that will allow sheriffs and developers to write tools that utilize this new hands-off landing queue.  We'd also be happy to have people work on the issues that come up when we take Autoland to the next level - auto-landing on a production branch.  To do this we'll want some automated backing out, bisection, and the ability to wait on getting patches reviewed before continuing.Another great area for someone interested in helping out Firefox developers is working on the TryChooser syntax and features.  There is a whole tracking bug dedicated to try_enhancements and most of those bugs are ones that can be worked on in a local staging environment.  It's a chance to get your feet wet with buildbot and our custom scheduling setup. Some of these smaller bugs would be short on time commitment and high on developer appreciation if you fix them. That can be a winning combination for a new contributor, I speak from experience on that :)So, if you're reading this post and you or someone you know is interested in dipping their toes into becoming a Mozilla contributor and these projects make you curious then come find me and we'll get you set up with a staging environment so that you can start fixing real world tools and automation bugs in no time.img, #cubbies-overlay{ -moz-transition-property: margin, box-shadow, z-index; -moz-transition-duration: 0.1s; -webkit-transition-property: margin, box-shadow, z-index; -webkit-transition-duration: 0.1s; } .cubbies-selected{ z-index: 9999; box-shadow: 3px 3px 8px -1px blue !important; cursor: pointer !important; margin: -3px 3px 3px -3px; } .cubbies-selected:active{ box-shadow: 2px 2px 5px -1px darkblue !important; margin: -1px 1px 1px -1px; } #cubbies-overlay{ position: fixed; z-index: 9999; bottom: 30px; left: 30px; box-shadow: 0 2px 3px rgba(0,0,0,0.8); border: none; } #cubbies-overlay:hover{ box-shadow: 0 2px 3px rgb(0,0,0); } [...]

Mozilla Seeks Program Manager for Open Web Innovation Incubator


Ok, I'm a little biased - full disclosure: I work for Mozilla.  But even if I didn't I suspect I'd be impressed with the amount of amazing innovation and hustle that Mozilla's community puts out towards making the open web more accessible to everyone.  Recent projects like Popcorn and Butter are changing the way we work with video on the web. Hackasaurus is reaching out to kids, getting them to move beyond consuming the web (read-only mode) to being able to build and design their own web experience (read/write).  Addons SDK, Open Web Apps, Browser ID, the list goes on and on and I'd better stop now or I'll lose you before getting to the good stuff.

Mozilla may be a huge brand but we're actually a relatively small group of people doing this work all around the world.  Which is why we now have WebFWD, a program to help innovators of the open web get a chance to hook in to the resources of Mozilla (space, mentors, public reach, food and housing, and more) to help bring their products to the world.  With this rolling program of projects Mozilla can be a driving force in getting more open web to the people who need them.

Right now we need a visionary and hard working Program Manager to become the leader of this movement.  I'm attaching the posting below, get in touch with p at mozilla dot com with your resume for consideration.

/// WebFWD - Program Manager

Mozilla, the organization behind the Firefox Web browser, is looking for
an all-star to join our new accelerator/incubator program WebFWD
( which aims to do for the open Web what organizations
such as Y Combinator, TechStars and Seedcamp have done for startups.

As the Program Manager of WebFWD you are charged with leading the
overall program - from designing and managing the curriculum, supporting
the selected teams locally as well as globally, working with our
ever-growing list of mentors and partners to organizing events.

If you're passionate about the Web, want to help people build amazing
products and are willing to roll-up your sleeves, then this position is
for you.

Primary responsibilities:
* Design and manage the curriculum for both the Fellow program as well
as the Bootcamp
* Work with and support teams in the program (both locally and remote)
* Work with our mentors and partners
* Coordinate community and press outreach on a worldwide basis
* Create and run events (locally and remote)

* Excellent written and verbal communication skills
* Experience with organizing and running events
* Experience working with startups, entrepreneurs, venture capital and
incubators / accelerators a huge plus
* Proven ability to work independently and in cross-functional teams
* Familiarity with Web technologies
* Passionate about helping people and solving problems
* Enjoys learning and teaching others
* Works effectively in a fast-paced, start-up environment
* 3-5 years of relevant job experience
* BA/BS, or equivalent experience

Try results to the bug(s) of your choice upon completion


The TrySyntax helper and TryChooser wiki docs have both been updated to reflect the new option when pushing to try where you can now ask to have your complete summary of results (and a link to the tbpl page for your revision) posted as a comment to the bug on completion.  Here's a live example to check out:
Sample comment in a bug when using --post-to-bugzilla in your syntax.

Now you have more control over how you get your try results and how noisy a try push is.

Please send comments and issues to the bug tracking this work.  Thanks for trying it out!

A quick morning rant about "gender" and data collection


This morning I read that Google+ is going to make your name and "gender" required to be public if you want to participate.  This bothers me for several reasons:

Web sites and forms notoriously say "gender" when they mean "sex" and only put M/F or Male/Female as options. When this type of choice is required but called "gender" it erases many people who do not feel that those options cover their gender since that is actually something way more mutable than your assigned sex at birth.  Solutions: Call it "sex" which is really what those two categories are or don't make something that is not in fact binary into a required choice of two options.

Google are so proud of being all scientific and data driven and I'm frustrated that they would not take the opportunity on their new potentially game-changing social platform to re-vamp data collection. Don't they have the processing power to allow people to put in whatever they like as "gender" and let the power of the search sort things out in the end?  If a small number of people want to put "jedi" or "dog" let those people find each other!  Who cares if there are some people who don't feel like Male/Female defines them?  Why Google? Why do you want to act like two boxes can cover the breadth of human experience as it relates to gender in this world?  Why can't you innovate on the small things as well as the big things that affect human interactions?

I'd really like to see a shift in how we collect data where there is more trust that the user knows who and what they are and that they want to share this information at their comfort level and that those on the other side, let's call them advertisers (cause isn't that what it all comes down to?), be the ones to deal with the outliers and uniqueness of human experience instead of trying to bash everyone into a two-party system.

Sidenote: When I have collected data recently for PyStar and allowed the gender field to be a text box I have found that the expected percentage (98%) of people entered "typical" information like woman, girl, female and that those who needed to express a different response appreciated the ability to do so by entering something else.  Leaving this field to user input choice did not result in a messy, chaotic list of random words or unidentifiable descriptors.  I fear not that most people will suddenly start to be something else when given more autonomy on forms.

Tree Closing Downtime Notice - 4am - 8am PDT Thursday June 16, 2011


Trees will be closed for downtime so that we can land the following:

1. -- Fix time on dm-wwwbuild01

2. -- Set journal_mode = WAL for dirty places profiles -- This mean new performance numbers will start on Thursday morning after the downtime

3. -- Run ANALYZE on dirty places.sqlite files --  This mean new performance numbers will start on Thursday morning after the downtime

4. -- reboot the DNS and DHCP servers in scl1 -- Rebooting these servers has been shown to burn builds in the past, requires a short (~5min) outage to reboot these servers to allow updates to take effect.

5. -- change LDAP to see if that speeds up mercurial -- This change should be entirely transparent.  Hg processes that are running at the time that the change was made will have already loaded the NSS LDAP module and will continue to use it until they exit.  The only issue to be aware of is that changes to hg access (group membership, or the creation of a new account) will not automatically propagate to the hg servers the way they do now.  If any hg access changes need to be pushed urgently, we can do that manually.

If anyone has a reason not to proceed with this downtime, please let me know.

Thoughts on cultivating an "Everyone is Remote" attitude


As I write this I am working from Paris and our team timezone spread looks like this: Rangoria, New Zealand: UTC (+12) Bucharest, Romania: UTC (+3) Istanbul, Turkey: UTC (+3) Paris, France: UTC (+2) <--- ME! Ottawa, ON: UTC (-4) Toronto, ON: UTC (-4) Philadelphia, PN: UTC (-4) Clifton Park, NY: UTC (-4) Chicago, IL: UTC (-5) San Francisco, CA: UTC (-7) Mountain View, CA: UTC (-7)I'm going to go out on a limb here and say this: Release Engineering does a good job of working remotely with each other. We are 15-16 people (with a few more contractors/fte on the way) and it doesn't matter where you live for you to work with us. Here we are in our meeting yesterday: Releng Weekly Meeting - June 2011Quite the impressive Brady Bunch layout, right?Here's what we do that I think works well for working remotely:* We meet once per week as a whole group on Mondays. This starts the week off with a status update on our major projects and also a chance for individuals to speak up about anything they're working on that they'd like people to be aware of.* We are always having conversations in IRC amongst ourselves and with others in several channels. We use #mozbuild as a backchannel for our inter-team discussion, #build for access to a larger group of fellow Mozillians (like philor, Kairo, and ted for example, who often need to liaise with us), #developers is also a place we frequent and then there are some IT/mobile/QA/release-specific channels we hang out in as needed. I think this helps us have a presence in many areas of engineering/dev/IT and even with some of the non-technical teams at Mozilla where inter-team communication needs to happen. It keeps us in the loop on what various teams are up to and also provides the IRC equivalent of being able to overhear water-cooler chat and participate as well.* We keep wiki pages for most everything. From "how-to" pages for our own release process, automation details, and project planning all the way to pages for outside-releng folks like the Try Syntax. While I find wikis frustrating the minute the information is out of date, the fact that I can update them and find them in my awesomebar quickly when I need them is very valuable to me.* We email our group with important notices and changes to how things are done. There are not often times when someone will say "Oh I didn't know about that" and the response is "It came up in the hallway when I was talking with so-and-so". More often than not, the person driving a particular upgrade or change to current practices will send out an email to the group with details of : a) what the change is b) what it means going forward c) how the message has been disseminated to a wider audience (if needed) and finally d) where the wiki pages (and bugs, if needed for reference) can be found. This allows any of us to find the information N time units later when the change actually comes up in your daily work and you're wondering "What was I supposed to do when trying to use the new X again?"* We all meet up face to face approximately once per quarter. Twice a year for Releng work weeks and twice a year at Mozilla all-hands/summit gatherings. We take these as opportunities to discuss larger topics with lots of brainstorming, whiteboard scribbling, and animated opinion-sharing. Notes from meetings like this turn into wiki pages (often during the meeting itself) and those can become specs for projects/bugs to carry the work that needs doing to the next level.I think that gives a good idea of our team practices. Now here are some thoughts I've been having about lately with regards to working remotely in Mozilla as a whole. It helps that I'm currently [...]

Use Try? Read this.


Two updates to Try are about to go into effect which enforce asking for what you want using the try syntax and configuring how much email you want to get with your results.  Read more below.

Bug 661409 - Now that this has landed, a push to try only generates email about a particular try builder's results if it does not succeed.  You can adjust this to be more verbose by adding a -e/--all-emails to your try syntax if you miss getting over all those emails, or you can just shut off the emails completely with a -n/--no-emails in your commit syntax. Note that you must be using the "try: " syntax for these email flags to be picked up which leads quite handily to...

Bug 649402 - Try syntax use is about to be mandatory as soon as this bug is fixed and the hg hook is enabled on the try repo. We're doing this to encourage developers who use try to take an extra moment and request only the resources they absolutely need on their push.  This should reduce the test/talos load that has been increasing wait times across all branches during busy periods.  One additional psychological change is that the "try: -a" syntax has been removed and in order to ask for a mozilla-central matching run you must be more explicit: "try: -b do -p all -u all -t all". I've updated the docs to reflect this change as well as the TryChooser syntax helper webpage. We're really not trying to make your life harder with this change, approximately 50-60% of pushes to try currently use the try syntax and if you push to try without it you will get a helpful message pointing you to docs and syntax builder.  Check with #developers for tips and tricks from the folks who've been using this since the beginning, I know they have many including using the newly-minted Mozilla-Inbound repo where a push will get the complete set of tests/talos if you'd like to let your patch bake for a bit after doing a selective try run.

Update on the Auto/Assisted Landing System


Almost a week since the post introducing the design attempt for auto/assisted branch landings via Bugzilla and Try and guess what? We re-wrote everything!

The details are in the wiki, bugs have been filed, code is being written.  We are working on making this system use a message queue and also see if we can work with mozillapulse to get information on bug changes from Bugzilla.

I'd love to tell you more about it but you can read the wiki and I'm excited to get back to my SchedulerDBPoller component.

Assisted/Automated Landing - Designing the Systems


Ehsan's blog post wishing for assisted landings on mozilla-central started a lot of people talking about this being a very desirable and useful tool for developers, where they could set a flag in Bugzilla and then be free to do other work until the results of their push were posted back to the bug. As part of enhancing the Tryserver I was already working on a way for users to signify in their try-syntax that the results of the push should go to the bug and these two ideas started to fuse into a dreamworld where someone could attach a patch to Bugzilla and have it be tried and pushed to trunk all with some magical bot automation.After doing a very short survey of developers and their try usage I have observed that there are two very different stakeholders here and both of them need separate-but-related tools: DevelopersThe Bot (automation) ease of usebetter reporting (less email anyone?)option to post to bug(s) *after* a try run has indicated successqueuing of patches culled from a flag in Bugzillaautomatically apply to tip of repopush, and report back with results After soaking in the survey feedback and a first attempt with a whiteboard yesterday, I woke up this morning with some clearer ideas on how to take a first run at creating this system.  It involves creating several new tools, one new database, and enhancing our existing buildapi.New tools for Developers:Adding more Try syntax options: include list of the bug(s) that you would like your try results posted to (however many make for a complete run on your push, this can be one linux build or a complete ~186 builder try: -a buildset) turn off email notificationsAdding functionality to the self-serve api view for a revision (eg: that will better show your results for that push and provide a button which will post the patch(es) to a specified bugAuto-landing from a bug in Bugzilla using the [autoland-try] whiteboard tag where any attached patches which are not obsolete, and have nothing set for 'r' are applied to the current tip of mozilla-central, pushed to try and those results are returned to a comment in the bugNew tools to Automate landings (bot or script):Crawl Bugzilla for bugs where [autoland-$branchname] is in the whiteboard and automatically push to tip of named branch, get the results, and return them to a comment in the bug (stripping out the whiteboard tag on completion)bot will grab all non-obsolete, r+ patches (if $branchname != 'try') interdependent bugs will not be handled in this first swipe at a working systempushes will have autoland-$bugnumber as the reason for the build in schedulerdb so that the results can be watched for, aggregated, and reposted to the bug on completionWatch results coming back for one or two oranges (we can set a threshold) and re-triggers those, watching for the second set of results - to attempt catching intermittent orangesBackout patches where even with a rebuild on an orange, there still remain orange resultsLDAP authentication checking for bugzilla patch author -> hg commit permissions and being able to ensure that only people with the right credentials can trigger automatic landings. This may mean checking the reviewer too before allowing a patch to be applied & pushed.The next step is to get this design organized into bugs so that we can parcel out the work involved and start testing/completing segments and features as we work towards the whole. We have a RelEng intern this summer, Marc Jessome (Another Canadian in RelEng!), who will be doing a lot of the work between now and the end of August. [...]

A PyStar Supernova in the Sky


The first Bay Area PyStar event has come and gone. I'm finally getting a moment to regroup and ponder all the trial and error of being the organizer of this event as well as having time to look at some of the statistics we gathered. Just from an organizing perspective here are a few items I'd like to share about the process.Things to do differently next time:* When creating the Eventbrite event, add questions like "What level do you want to learn at?" "Meat or Vegetarian?" "Operating System?" to the registration so there's no need to send out blanket emails to attendees to try and get that information after the sign up.* Only do one day workshop instead of Friday night installation and Saturday workshop. I think that for many people the setup could have been done in the first hour and the rest of the day been spent learning instead of having a night session that only is needed by a handful of people. * Have the teachers/assistants already assigned to a particular level of instruction - prepare topics, tutorial materials, and class size ahead of time so that on the day of the workshop there might only be a handful of late arrivals to place and the other attendees will already be set up in the right learning level as requested in the sign up.  Things that really worked this time:* Eventbrite! They have amazing tools, stats, emailing options, charts, and also a way to see where your sign ups come from which showed us that we got a TON of views from Tweets which apparently was an impressive number (I am told by one of our attendees who is an Eventbrite employee)* Mozilla! By sponsoring the event - providing the space and food - being able to let people/groups spread out and work in our various conference rooms as well as having lunch on site was very much appreciated by attendees (and of course by me!)* CodeChix!  This peninsula-based group of women coders accounted for 30% of our attendance and also netted some teacher/assistants for the workshop. CodeChix co-sponsored the event and helped get word out as wellAttendanceThere was something odd happening with the Eventbrite signups. In a couple of short bursts, a ton of tickets were being snatched up by names that seemed slightly suspicious. Now the event has passed and I've checked in all the attendees as well as accounted for the no-shows (almost all of whom took a moment to send in their regrets so the tickets could be freed up for another person - very sweet!). It looks to me like about 40% of our attendees were fake accounts. Julie (who works at Eventbrite) and I took a look at the numbers and she's kindly offered to look into it further to see if there is indeed something fishy happening. All that aside, we had 47 people! That feels like great attendance to a first workshop, on a Saturday, in Mountain View.Speaking of Mountain View - we had attendees come from all over Northern California. I love this view of how spread out geographically we all were:This graph is useful for seeing how my own promotion attempts were successful.  The original spike of page views is obviously when I first announce the event link. CodeChix, Baypiggies, and Devchix were the mailing lists I sent emails to with the link. While that got the ball rolling, it was the tweets and emails sent out almost 3 weeks later - a week before the event where the event got lots of attention.  It probably helped that PyStar Minneapolis was happening then too so #PyStar got lots of tweets (sorry to the person who's twitter nick is @pystar).Can I just say that I am so thrilled with the amount of people who volunteered to teach/assis[...]

Captain Destructo Breaks Everything


Alternate title ideas: "It's not all s/Tryserver/Try"  or "What I should have done, and didn't"I bet you get the point by now. Today I caused a fairly lengthy, unnecessary downtime on Try.  Now that I'm writing this, things are under control again and there's a few small niggly bits left but nothing that will keep me up at night.It all started with a bug about graphserver posts from tests not getting through because they were looking for MozillaTry (the tinderbox name for Try) but instead the graphserver only knew about Tryserver (the branch name for Try) and nothing was using Try (except the repo for Try) which is what it ought to have been doing in the first place.Now that I've been adding a lot of project branches in a short amount of time, certain things have become more streamlined and so I felt that the best option was to go through and rename Tryserver/MozillaTry to Try everywhere so that from the repo going forward, everything was the same. This has been working extremely well for our project branches and helps make setup a snap.Here's where it gets all broken. I approached this bug with a quick swipe at this problem was superficial and ended up causing some preventable burning.  I shall now list for you (and future me) what I did and what I should have done:Did: hg rename on configs for desktopbranch configs for s/tryserver/tryupdated graphserver branch name to Trya quick downtime window from 10am - 12pm in order to prevent builds from getting split into two different upload dirs Should have done: hg rename on configs for mobilegrep of buildbotcustom for "tryserver" as we have special casing for it in several fileslog uploader and post_upload scripts to make sure everything about the try build was going to the right place updated the dir permissions on ftp for the new upload location and ensure that the archive is on nfs mountedited cronjobs on staging to catch the new try builds updated graphserver machines table for each try platform's builder namemore notice for downtime, with a 4 hour window that would have allowed a test push to make sure everything was wired up correctly  updated the treeclosure hook to include the new tinderbox page Some of the things I should have done didn't have an impact on the burning/try closure but it's fair to say that if I had done a staging round of all my plans first I would have caught more of the obvious things that I missed. I would have then planned the downtime better and been prepared to ensure the disturbance would have been minimal since this was, after all, a really low priority bug.Aki told me that he had a manager who said "you don't learn til you break something".  Well I broke everything try-related today and here's hoping that I have learned something because the stress of this whole day is not something I want to experience often. It's that feeling you get when you realize you've started something that you can't back out of and there's no way to go but forward, even though everything in front of you now appears hopeless and messy.So here's some lessons to take away:Staging is not to be underestimated even for just renaming things that are already workingTaking the time to search with grep/mxr and find the terms you are replacing before starting the upgrade in production will help find wiring you might have overlooked in your preparationsPrepare more thoroughly and have a clear idea of the env. you started in and what it will take to have that env. back when you're done. Leaving dangly bits is not ideal. Happy Friday. (and many thanks to Aki)[...]

Hey BBC would you like to know how releasing software works?


Dear BBC,Today on the front page of your technology section you said that downloads for Firefox 4 have been lower than they were for Firefox 3 and that: The lower figure may be explained by the widespread availability of pre-release versions of Firefox 4 in the months ahead of its launch.First of all, you forgot that we've had 3.5 and 3.6 between those two and so we now have users spread out a bit across versions. Second, here's an overview of how we're organizing the release of Firefox 4:We put out the RC and picked up users from outside of our usual beta testing pool in order to give our final candidate some solid tire kickingFirefox 4 went live but our users on 3.5 and 3.6 are not offered the update automatically yet, they must "Check for updates" in order to be asked if they want to upgrade to Firefox 4 Once we have more coverage of the new release for a couple of weeks and are even more confident that we've got an amazing browser out there we will turn on the Major Update notification which will offer our 400+ million users the chance to come on up and experience the next level of the webAccording to W3C school's stats(which are measured by visits to their site) the browser distribution of their visitors looks like this: 2011 Internet Explorer Firefox Chrome Safari Opera February 26.5 % 42.4% 24.1% 4.1% 2.5% January 26.6 % 42.8% 23.8% 4.0% 2.5%2011 Total FF 4.0 FF 3.6 FF 3.5 FF 3.0 Other February 42.4 % 1.9 % 35.8 % 2.9 % 1.5 % 0.3 % January 42.8 % 1.5 % 36.1 % 3.1 % 1.7 % 0.4 %What this says to me is that our more than 8 million downloads since yesterday morning PDT only shows us how many people are paying attention to the fact that Firefox 4 has launched and is available for download. It's not representative of our 400+ million active daily user base (the people who just use the browser but perhaps don't read your blog or mine).  These people will soon learn about Firefox 4 through their browser's update notification window. We'll be seeing a spike in downloads in a couple of weeks and I hope you'll report on that.[...]

Bay Area Video Coalition - Teaching Open Video Part 1


Last night was the first meeting of The Factory and Mozilla. The partnership if a result of work between Mozilla Drumbeat and Web Made Movies. Ben Moskowitz and Brett Gaylor invited myself and Atul Varma to what is to be the first of three sessions helping teens learn about the budding open web technologies that can be integrated with video.About 13 kids streamed into a computer lab at 4:30pm and we began with some introductions detailing who we were and why this stuff is interesting. The group were very engaged and eager to dive right in with whatever we had for them. So we started with Atul's Web X-Ray Goggles so the students could see what exactly the web was made of. The idea was to just grab parts of whatever websites you liked and change them right on the page so you could see how easy it is to "hack" the web. Some of the teens went even further and started building their own pages with Etherpad by grabbing snippets of code from sites. About 4 kids said they had done a View Source on a page prior to this class and 30 minutes into this workshop they were all doing it like pros.Once they had a chance to remix a web page we moved on to the next exercise which was to select 4 popular sites of their choosing (Etherpad democracy!) and those sites were printed out on paper, the teens were split into 3 teams, and each team did paper prototyping of a new site using elements from the originals. I was very impressed with how the students took the idea and ran with it. Each team worked fast (they had 7 minutes) and no one was hanging back keeping their opinions to themselves. The teams produced 3 new site mock-ups that each had a very simple look, with a video as the prominent element on the page but they also took time to add site navigation and social media integration by putting Facebook and Twitter in the sidebar or footer.In the last 30 minutes of our time Brett and Ben demonstrated Butter also explaining how very "caveman" the technology is right now. With only a glance at the interface and a basic explanation of how it's wired up the teens jumped right in with suggestions and ideas about what they would like to be able to do:* Hide popcorn elements when nothing is showing in them* Be able to zoom in or out on Google maps while the video is playing* Click on something in the video (example: coffee mug) and have it trigger an eventBen made a really great point about how it's also important to look at something like Butter and think "How can you go beyond the interface?". How do you make your story more interesting from the beginning knowing you can use this tool throughout instead of just tacking on events and additional information to a completed video that is done in a standard format?We'll be working with them again tonight, with chunks of a film they made last summer called "The List". More updates and more potential bugs and feature requests coming soon.Also, if anyone is planning to teach a class like this you might want a few things in a "kit":* Portable printer (and paper)* Scissors* Tape or glue* Handout with links to the tools/sitesJust to save some time :)[...]

Volunteers needed for upcoming HTML5/Open Video tutorial


I'm hoping if you're reading this that you might be interested in volunteering this coming Saturday to help 12-16 year old girls at the upcoming Dare 2B Digital conference learn about HTML5 and open video.  There's more information and background on what's happening on this wiki page.

Two kinds of volunteers needed:

1.  Someone who is in the Bay Area and available this coming Saturday from 9-3:30pm to be on-site with us in Mountain View at the Computer History Museum and will work hands-on with the girls to demo Miro Converter, Universal Subtitles, and a little bit of Popcorn.

2. Anyone, anywhere, who can do translations to any language and who is available on Saturday anywhere in the 10:15am-3pm PST window to do some 'live' subtitling and show the workshop participants how amazing the universal subtitles project is.

Please get in touch if you are interested/available. Or sign up on the wiki.

Thanks in advance!  I will be posting any demos, workshop materials, and an update post-event on how it went.

Automated Try Results Posted to Bugzilla - A request for input on what the comment should contain


Lately I've been working on a a script which can check your try syntax for a bug number and a setting asking for --post-to-bugzilla.  If you've provided both, your try server results can be posted directly to the bug.  This is just part of a larger project to have patches submitted on a bug get automatically tried out, results posted, and at some point down the road they could even be pushed to trunk after a successful try run so "look Ma! No hands".Today I have a script running in staging, polling for completed try runs and doing dry runs of posting to bugzilla in log format only so I can keep an eye out for unusual output.  Already this has shown me a couple of bugs to sort out, and I anticipate having them ironed out very soon.  However, before this lands I would like to get some feedback/ideas/suggestions on what the output to the bug should look like and there could be a couple of options even, with a setting in your try syntax.In my early testing I posted to our landfill bugzilla and here's what a couple of results looked like:Lots of success looked like too much infoSo I took it out, and only printed the warnings and failuresWhich works on small runsFrom what you see here, I'm sure you can imagine what it would look like if 145 builds all had warnings/failure combos.So - what do you want to know in the bug? Let's keep it simple, ok? We can add more later and it's important not to create a bug-spammer here that folks will clamor to turn off soon after it goes live.Off the top of my head, and after talking with Catlee today about it, I think it should look like this:Try run for $revision with the following comment: $try_syntax line S:# W:# F:# (results total) builds complete from N total requestsS:# W:# F:# (results total) tests complete from N total requestsFor more information please see$revisionThis gets you a quick glance at total builds/total requests so you can see that everything is accounted for, and where things might have gone wrong in builds vs. tests but doesn't list the failed/warning builder names so you have to follow the link to get them.  Maybe there would be interest in printing what your try syntax request triggered but I'm not sure that's useful in the bug reporting even though it's requested for when a developer is pushing to try. What do you think? It would be possible to break down the results further by platform instead of or as well as build/test.  Any ideas on how to get that much information across without making a bug comment too verbose? All input appreciated and considered, I'll be trying to land this in the next couple of weeks so comment here, ping me in IRC (lsblakk), or comment in the bug.[...]

Try Server Road Map - Q1 2010


The other day this post by Google with slides detailing their Chrome release cycle speed up was going around and it mentioned how try and CI were key to their success. It got me thinking that it's time for another update about the upcoming improvements our try server automation. Most of my Q1 work will be on the try server, with some time on Fennec Beta releases, and a bit of time also working on making it much easier to spin up disposable project branches.The road map image above shows how there are three areas of focus and here they are now in a more detailed list with bug number attached:Improving Current AutomationBug 617321 tracks adding two new try buildbot master instances to our other buildbot-masters in the Santa Clara colo.  This gives us flexibility to have rolling downtimes on try (as we already have on the other masters) where we can update things behind the scenes and it also helps by adding redundancy to the try automation in case of a colo outage.Bug 580346 is almost done and it adds xserves to try which gives us some faster macosx building power, that along with about 40 ix builders for win/linux builds will crank out more try builds faster.Bug 594236 is key to getting TryChooser syntax turned on as a default.  With an interactive hg prompt on push to try, you should be able to select what you want and/or have your syntax validated.  Khuey started something to do this and if anyone is up for taking it to the next level before I can get around to it, please please do.Add FeaturesBug 430942 is what I am actively working on today and the rest of this week, I'm about to have a second draft ready for review and I expect you can watch for this to land in the next few weeks. With or without try syntax, you will be able to specify the bug number in your push comment and have the results of your try run posted as a comment on that bug.  Hopefully this will help out development in letting people know where something is at if you are away when the results come in. It's also part of our old bug (pre-2010) smack down goal so finishing it will be one more step toward that carried-over goal being met. Bug 615705 is tracking a few more tweaks to the try syntax that will give users more flexibility in the syntax and choice about what to build.Bug 421895 is another old bug and Chris Atlee is close to getting it up for poking at. It provides a way for users to cancel their own try build requests without having to ping RelEng.Bug 621681 addresses having better threading/headers since the current headers only help with threading for some clients. When I first wrote it I was testing with Thunderbird, where it works as intended but apparently Gmail and other clients need some help.FutureLooking to the future right now we have bug 625464 which talks about setting up something to scan bugzilla for a flag on attachments that will trigger an automatic try run with that attachment and either tip of trunk or perhaps a user repo, which would require the other main future bug, 625463. With the ability to poll and run try on hg.m.o user repos we can have project branches (temporary branches that are loaned out to devs or teams to work on a particular project) toggle a setting that would have their pushes to the repo get run through try instead of the main mozilla-central automation.  This could be handy when you want to limit the machines you are building/testing on with the TryChooser syntax.I[...]

Please use TryChooser


Recently there were some improvements to the trychooser and the landing of those changes led to a couple of bugs[1][2] being discovered and quickly fixed.  It is thanks to those who are regularly using the trychooser that we are able to find bugs quickly and also continue to improve the tryserver.

Right now there are over 350 backed up test/talos requests for the tryserver and when I checked our report for trychooser usage it shows that the average number of users pushing with a try syntax has fallen below 50% where it used to be closer to 60%.

I encourage you to please use the trychooser syntax as much as possible. If you do not need every single try result for your patch, do not just push to try and use up all the resources needlessly. Take a moment to insert some try syntax into your commit message.

See for details and for a simple try syntax builder.

Thanks in advance.