Last Build Date: 2013-06-01T10:57:16-07:00
A couple of days ago we announced Watchman on the Facebook Engineering blog.
Watchman watches files and records information about them as they change. You can arrange to trigger build or test steps in response to changes in matching files, but the main the reason that we built it was so that we can instantaneously query file status for a set of files.
Watchman maintains a view of the filesystem that is kept in sync using kernel filesystem notification facilities. This view is indexed so that we can quickly return information about the watched portions of the filesystem and also query the set of files that changed since a given point in time.
since index is the heart of this; we maintain a linked list of file nodes, with the head representing the most recently changed file. As new files are observed they are inserted at the head of the list. As existing file changes are observed they are unlinked from their former position and inserted at the head of the list. This keeps the list sorted in time order without having to employ a traditional sorting algorithm; it’s very fast to update the list and this is important in the face of a high volume of change notifications.
Finding the set of files that changed since we last looked is then a simple walk from the head until we reach the time in question. The cost of the search is proportional to the number of changed files since we last asked.
We use both the triggering and the querying features to accelerate our build processes for the main Facebook web site. They are important to us because we have such a large number of files that it is impractical to manually maintain a traditional static build recipe; we need to crawl the tree to determine the nature of the build. Since the tree is so large, and due to other factors in our dev environments, we also run into issues with the filesystem cache going cold; we need to minimize our I/O profile so that we only visit the files that we truly need to in order for our builds to run fast.
This year one of the changes I made was around my eating and exercise habits. I managed to lose 30lbs in just under 3 months. I wanted to share how I did this because it seemed much easier than I thought it would be.(image)
2012-11-01T00:00:00-07:00I’d promised a couple of folks back in Maryland that I’d keep updating this blog to let them know what I’m getting up to. I’m a little overdue on this, but it does mean that I get to write a more substantial update. (I post much more frequent but smaller updates on my Facebook timeline!) As I write this entry, Juliette, Xander and our two dogs are having their final sleep in our house in Maryland. It’s taken a bit longer than I’d hoped, but in the morning we’ll be reunited and can really get this new life going. July July had me bouncing around a couple of different accomodations and gave me a bit more insight into the surrounding area, but it got a bit tiring towards the end of the month. My car was shipped out and arrived here after a couple of weeks. Was great to get it back :-) Work wise, I officially joined the Dev Tools team at Facebook. Our area of focus is general developer efficiency; Facebook engineering is very much geared up for rapid iteration and our group works to ensure that we have appropriate tools and infrastructure to enable those efforts. Dev Tools has close ties to the Release Engineering team (which is headed up by Chuck Rossi) and so our summer off-site had both teams go to a range for some instruction with rifles. I had a lot of fun doing this, despite catching some hot brass on my neck (scarred up quite nicely!). Juliette and Xander joined me for a while towards the end of the month and we went apartment hunting. August I moved into our new apartment. I’d managed to coordinate getting some basic furniture, supplies and internet connectivity to be delivered on the first day. It was a bit of a spartan existence for a while because the bulk of our furniture was still on the east coast. At Facebook, We had an Infrastructure group hackathon. I can’t tell you what we hacked on, but it was fun to participate. We watched the movie “hackers” through part of it, which was great for a few laughs. September Space shuttle Endeavor flew over the office. Through the previous months, I’d been living a bit of a batchelor lifestyle. I’d spent a bunch of time in a bar in Palo Alto called The Old Pro, which apparently has a bit of a long history with Facebook. This was good for me; where I’d lived in Maryland didn’t really have any kind of night life and I was playing catch up. Moving into September though, my friends and I were slowing down our Old Pro visits. Since I was home alone pretty frequently, I needed to find something to keep me occupied in my downtime from work. The apartment complex I’m in has a fitness center and I eased into a cycling routine using the stationary bike. I bought a wifi connected scale and a fitbit to track my weight and activity. This was very motivating for me personally and I highly recommend it if you’re looking to shift some weight and keep it off. October October has gone by quite quickly. It started to cool down a little bit. I’m still impressed at how warm it is; I do wear a jacket in the evenings but the days are often warm and sunny. Love it! My fitness routine and revised diet started to show results; I had to buy smaller clothes and it feels great! And that brings us up to the present. I’m happy with the changes I’ve been making in my life and excited to see what the next couple of months will bring! [...]
2012-06-25T00:00:00-07:00Earlier this year I made a difficult choice: to leave Message Systems and join Facebook. I moved countries to work with OmniTI/Message Systems back in 2004, and I’ve worked with the team for almost 8 years, seen things grow from a couple of engineers and a sales guy to a company with multiple offices in the USA, Europe and Asia/Pacific. In my time with the company I’ve learned a lot about many things, worked hard with some great people and had good times. So why leave? It was time to try something new. My day-to-day had settled into a fairly repetitive and steady pattern and there wasn’t a huge amount going on in the neighborhood where I lived; rural Maryland isn’t exactly a hotbed of activity! Add to that the oppressive humidity in the summer and the snow in the winter, I felt like I was never really all that comfortable getting out of the house. So it was time to try something new, and the time was right to give Facebook a shot. My last day at Message Systems was the start of June; we parted on very amicable terms and I’m going to miss the team: Facebook After talking with a couple of different folks about possible futures, I’d settled on Facebook as my next great adventure. While the hiring/interview process was a little bumpy, I had a lot of excitement around the sorts of things that Facebook are doing, and it really seemed like they had a lot of fun working hard. I’ve been a full-time employee of Facebook for just shy of a month, and I’m blown away. The on-boarding process is well structured, fun, and informative. Facebook believes very strongly in enabling you to get stuff done with minimal obstruction and there’s a great vibe on campus. We work hard and play hard; there’s opportunity to blow off steam in many ways, and this is actively encouraged by the culture to help keep folks energized. I’ve had code deployed to the main facebook web site within days of joining, expanded a high performing internal service in C, changed RPC interfaces, worked in Xcode, changed some Flash code, learned new build/push infrastructure and tooling and more. The scale of our operations is staggering and amazing, and the pace is rapid. I’ve commented a lot on my Facebook timeline about the perks and activities (which often include alcohol!) that keep it feeling fun. I’m sure my family and friends think I’m partying all the time, but that’s because I can’t post details of the work I’m doing and that biases the results. The campus has modern fittings and has a kind of industrial engineering-in-progress feel to it that helps reinforce the culture: the not-quite-finished look reminds us that our journey is not done and that we’re continuing to iterate building and improving our services and apps. One of my criteria for the next big adventure was that the climate be more conducive to getting me outside, and California scores highly in this department: Moving So far I’ve been staying in temporary accomodation in Mountain View. The next thing on my agenda is getting the family relocated from Maryland to the Bay Area. I hate having to deal with the logistics of moving, but can’t wait to establish a more permanent base out here. As first steps go, off to an amazing start, but this journey is only 1% finished. [...]
This time last year, Chris wrote about blogging and how Twitter has reduced the quality of online discussions by shifting the focus from longer form discussions to shorter tweets. He was hoping that he could foster something of a blog revival, and did for a while.
This year Chris is hoping for more of the same, and invited a number of folks to join in to see if we can get some more momentum going.
I personally prefer reading blogs to reading things on Twitter:
I also like to blog, but have found it difficult to make time to turn out content of the quality that I prefer. I hope to change that this year, and blog some more. I’d also like to see you blog more too.
I’ve had a couple of folks ask me about the form validation I mentioned in my last post, so here’s the code.
It’s fairly basic IMO. This is public domain code; use as you will and at your own risk. You’ll be able to make a form that looks like this: