Subscribe: Jeremy Zawodny's blog
Added By: Feedage Forager Feedage Grade B rated
Language: English
barnes  barrel  bit  connect errors  connect timeout  connect  long  lot  master  max connect  mysql  new  oracle  time  work  yahoo  years 
Rate this Feed
Rate this feedRate this feedRate this feedRate this feedRate this feed
Rate this feed 1 starRate this feed 2 starRate this feed 3 starRate this feed 4 starRate this feed 5 star

Comments (0)

Feed Details and Statistics Feed Statistics
Preview: Jeremy Zawodny's blog

Jeremy Zawodny's blog

Random thoughts on technology, aviation, and life in general...

Published: 2010-02-25T17:20:32-08:00


I'm Blogging Elsewhere


I haven't written much here recently. And that's really due to a lot of reason, not the last of which is being busy and that we have a wonderful new kitten named Bear:


The other issue is that I've becoming increasingly unhappy with my blogging tools, the custom hacks I've added over the years, and the required maintenance and upkeep.

So I've decided to give WordPress a try. You can visit my new experimental (likely semi-permanent) blog here: Jeremy Zawodny's WordPress blog

I may find myself posting stuff here from time to time. Maybe. But the reality is that I'll probably work on a mega export of this stuff into the the new blog and setup some redirects at some point.

When I have time.

Which means it may take a while. But that's life.

In the meantime, I have more efficient tools over there and am likely to publish more often. So check it out if you're interested. Or not.


Zip Line and Rappelling Adventure in Puerto Vallarta


A few weeks ago, Kathleen and I went on our first cruise together. It was her fifth and my first, so I got to experience a lot of new things. If time allows, I'll try to write about that in the future. But for now, I want to talk about one of our shore excursions. Our first day in port was in Puerto Vallarta. The weather was absolutely perfect: a few scattered clouds, light winds, and about 80 degrees with bright sun--a far cry from the storms in California. We disembarked from the ship and met up with a group who were all going on the same excursion. We were then taken to what I can only describe as a "speed raft" designed to carry about 20-30 people. We boarded the boat and were treated to a high speed ride of roughly 20 minutes across the bay to a small beach and dock area. The cruise across the bay gave us some view of very nice water-front houses. It appeared as though there are a lot of wealthy folks that decided to build vacation homes there. At the dock across the bay, we got off the boat and were lead a few hundred feet to a guide that instructed us to board one of two 4x4 trucks outfitted with benches in the back. The trucks took us on a 15-20 minute ride through the town and into the forest and up to a higher elevation, ultimately arriving at the "base camp" for the operation. There we were asked to leave behind anything we didn't want to get wet. That meant cameras, wallets, phones, etc. They repeatedly warned us that we'd be getting very wet. We were also told, much to our surprise, that we wouldn't need bug spray or sunscreen. Most of our time would be in the shade of trees. After depositing our belongings to a communal bag that would be placed into a secured locker, guides helped get us suited up with our harnesses, helmets, gloves, and other gear. Then we were lead to small set of benches where the guides were introduced in a fairly comical fashion and they spent some time teaching us three important signals: speed up, slow down, pull your legs up. We were also instructed on proper hand placement on the zip line, posture (holding your legs up a bit), and braking techniques. (I noticed during the introductions that there was roughly a 1:2 ratio of guides to participants.) After that was out of the way, we headed over to stables where each of us was paired up with a mule for a ride up the path that leads to the top of the course. The mule ride up was probably 15-20 minutes and ended near the base of the first rappelling platform. There we did some brief stretching, reviewed the signals, and had a chance to ask any last minute questions. Then it was time to climb a short hill up to the first platform. This platform, unlike others we'd see later, was large enough to handle the entire group, so we could watch the procedure for getting hooked on to the lines. One of our guides went first so we could see what everything looked like--and that it was possible to arrive at the other end safely. Ready to Go After the first guide went across (he was on the other end to unhook us and catch us if we arrived a bit too quickly), the line of participants stepped up the the ropes, one at a time. One would get hooked up, checked, and then was told to go. While one was zipping, the next person would be hooked up and released once the first one was safely on the remote platform. The first zip was reasonably long but a fairly shallow slope, so you couldn't get going too fast even if you didn't brake to slow down. I decided to see how fast I could get going before I felt like I should slow down. It turns out that it wasn't until I was approaching the platform and got the "slow down" signal that I needed any substantial braking. I did "tap the brakes" a couple times just to prevent myself form turning, since that's the only way to keep yourself facing forward as you go. I arrived at the platform, stopped, and was unhooked. "How was it?" asked the guide. "Awesome!" I said. I knew the rest of the day was going to be a lot of fun... From there we went on to t[...]

Paper vs. Screen


I arrived at the Knoxville, TN airpot a bit ago for my slightly delayed flight back to San Jose via Dallas. Since I finished my book on the flight(s) out, I stopped by one of the shops to do something I rarely do: buy print media. Well, I still buy books now and then. But today I bought a couple of magazines: Disocver and Business Week.

I sat in the Ruby Tuesday's at the airport and managed to read a few articles in Business Week. After a bit I noticed that I was reading stories that I'd probably never read on-line. My on-line reading tends to be a lot more focused and directed. Reading for at a relaxed pace for pleasure is a whole different feeling.

It makes me wonder what I'd do with a Kindle or iPad if I had one. Would I use it the same way I read on a computer? Or would it be more like reading a magazine?

Hard to say. I have no plans to buy either device, but in a few years when I replace my existing portable computing devices, odds are that I'll have something tablet-like.

I'm not sure I'd want to be in the magazine business a few years from now. I feel like it's following the "lead" (if you can call it that) of the newspaper industry.

Interesting times.


Remembering Barnes


Yesterday was a very sad day for us. Barnes, one of our four cats, had to be put to sleep after we took him to the pet hospital because he was having trouble breathing. A bit over a year ago, Barnes was diagnosed with Diabetes and he'd been getting regular insulin injections (2 per day) and vet visits. In fact, his most recent visit (3-4 weeks back) was very good. The doctor was happy with his progress and he generally seemed peppier. His blood test results were all very positive. But yesterday when we noticed he seemed to be working harder to breathe, I knew something was very wrong. He was placed into a 100% oxygen environment to make is breathing easier while they conducted some tests and gathered information. After some X-rays, we learned the extent of his troubles. He had an enlarged heart, fluid in his lungs, and cancer. In a way, it's good we didn't see this coming. His brother Noble seems to be taking it better than we are, and they'd been together their entire ~12 years. His newer brothers Timmy and Thunder will surely miss him as well. His favorite activities in the world were eating chicken (and fish), exploring our catnip plants and toys, getting belly rubs, and napping on his favorite fuzzy blanket. Barnes was the most loving and strong cat we'd ever known. He was probably hiding his illnesses for a long time, hoping for a few more belly rubs with us and visits with the catnip plants we grew. We'll really, really miss him. Here are a few of the pictures and videos we'll use to remember him... Barnes as a Kitten Barnes and Noble as Kittens Banes on the Bed Barnes Stretches Barnes and Noble Cuddle More Recent Barnes Photos Barnes on the Floor Barnes Barnes and Noble Barnes and Noble Videos of Barnes Barnes with a Catnip Toy Our cats eating Chicken Please remember to get your pets regular checkups with a good vet. They deserve it. (comments)[...]

Sphinx and Gearman: A Distributed Computing AH-HA! Moment


A week ago I decided to finally get serious about putting gearman to use for search indexing. I had been batting the idea around in my head for a long time (too long, really) but figured I should just write the code and see what happens. It took less than a day to get a prototype working in our development environment, but the end result made me very happy.

Today, in our production deployment, when a sphinx cluster pulls new content to index, the master does all the work. It fetches the new and changed postings, massages them into the XML format that sphinx expects (and makes a lot of small changes along the way), invokes the indexer, and makes the new indexes available for the slaves. The second step is usually the most CPU intensive. Processing the raw data into XML involves a lot of other tweaks and changes that are very specific to Criagslist.

What I did was turn that into a gearman client/worker pair. The client (or master) simply submits processing tasks and then waits for each of them to complete. The workers fetch the data from the master, transform it, and make the transformed data available. When each task completes, the master grabs the transformed data an informs the worker that it can delete the file.

So instead of being stuck at using only the 4 CPU cores on a single box, I can run 4 workers on each of 3 machines and get 12 CPU cores involved. The end result is that I have a solid foundation for a system that can easily scale to many machines. AH-HA! Linear scaling rocks! So does relatively seamless distributed computing.

As time allows I'll have to work on deploying this in production.


On the MyBlogLog Shutdown


Marshall Kirkpatrick is reporting that Yahoo! will shut down MyBlogLog next year. Well, color me unsurprised. The service has languished for years. I removed it from my site a long time ago. It made me a little sad to do so, but it was just slowing things down and not really "adding value" as they like to say.

It's sad because I was involved in the MyBlogLog acquisition at Yahoo! and believed in what they were doing. I worked to help get the team on board, nag the right people to make sure they got reasonable hardware on which they could grow, interviewed their first post-Yahoo engineer, and made the trek up to the Berkeley office a few times a week to help them transition into Yahoo and work on plans.

I genuinely had high hopes for what MyBlogLog could do both inside and outside of Yahoo. But as I wrote in Watching Yahoo's Transformation:

MyBlogLog has all but died on the vine, right? Is there anyone left of the original team of 5 or 6 engineers still working on it? No, I think it fell victim to Yahoo's larger social strategy. FAIL.

On the one hand, it's sad that our collective time was wasted, but the members of the MyBlogLog team have all gone on to bigger and better things outside of Yahoo. And I suspect everyone involved learned some important lessons along the way.


Trust Oracle? Why?


In a 10-point press release issued today Oracle has listed a series of "commitments" regarding their acquisition of MySQL by way of acquiring Sun.

I am not impressed.

As a former employee of a large Internet company (the largest at the time, in fact) that used both Oracle and MySQL, I'm utterly puzzled by this. I can't think of why we should trust Oracle to do right by the users of MySQL--especially the non-paying users.

You see, for years Oracle worked agressively behind the scenes to discredit MySQL and tried hard to understand how their customers could ever consider using such a "toy" instead of their flagship product. In fact, it was so important to Oracle that they offered some very substantial discounts to customers who were using MySQL and Oracle. In some cases the discounts were so impressive that their motivation was clear: cut off the opportunity for MySQL to grow and spread in such organizations. (Remember what happened to Netscape when Microsoft gave away Internet Explorer for free?)

The funny thing is that it really didn't work. MySQL was already a fast moving train with lots of momentum. And it was still accelerating.

It was clear that Oracle saw MySQL as a threat to their business. When they eventually bought Innobase (the company that makes the InnoDB storage engine), many of us got more than a bit nervious. That put Oracle in a position of having a choke hold on the one componenet that was critical to MySQL's future success. They could have just shut down development entirely. But that may have made their motives a bit too clear.

Since then they've continued to develop InnoDB. However, the pace hasn't exactly been agressive and their openness around that has left me (and others) wondering what their longer term plans really are. The few tidbits we get seem to be overly vague. Could they have been throttling development of InnoDB? Or not providing the same resources that MySQL (and now Sun) would have? It's hard to say.

But here's the thing that continues to bug me...

Back a few years ago when Oracle dismissing MySQL in public while working hard against it in private, I realized that they were simply trying everything they could to protect their crowned jewels: public denials and classic FUD paired with hush-hugh backroom deals.

Nobody has managed to explain, in even a mildly convincing way, what has changed since then. Why should we suddenly trust Oracle? Their crowned jewels are still threatened by MySQL.

Convince me.

See Also: Monty's appeal is selfless!


Weird Email People Send Me


I really have no idea what prompted this:

This is Rev.Willson and am interested in some of your Barrel. I need the Barrel in the specifications of 55 Gallon Barrel. Blue Plastic.So what will the total cost of the Barrel with the dimensions i gave and i need 100 quantities of the Barrel.So i will like you to go ahead and quote me the total charges of 100 quantities of the Barrel + tax without shipping charges included.Because i will be picking this up as soon as it is ready from your location.Please if you don't have the type of barrel that am interesting kindly email me back with the types that you have.And i will be very glad if you can also email me back with the types of credit card that you will accept for payment. Please advice. Thank You.

It's so random that it's funny.


Speaking at San Francisco MySQL Meetup


I'll be speaking Tuesday night at the San Francisco MySQL Meetup. Here's the blurb from the announcement:

Jeremy Zawodny will talk about Craigslist's current use of MySQL, where it's painful, and how things are being re-architected to make the pains go away. This includes hardware changes, sphinx, redis, memcached, and some custom Perl work.

Despite not living too far away from the SF MySQL Meetup venue, I'm a little annoyed at myself for never having attended. So this will be my first time and I'm really looking forward to meeting local like-minded folks.

Thanks to Mike and Erin for recruiting me.

If you're coming and have specific things you'd like me to talk about, drop me a note in the comments or on Twitter.


Fixing Poor MySQL Default Configuration Values


I've recently been accumulating some MySQL configuration variables that have defaults which have proven to be problematic in a high-volume production environment. The thing they all have in common is a network blip or two can trigger some very undesirable behavior.


If a client is having trouble connecting to MySQL, the server will give up waiting after connect_timeout seconds and increment the counter which tracks the number of connect errors it has seen for the host. Then, when that value reaches max_connect_errors, the client will be locked out until you issue a FLUSH HOSTS command. Worse yet, if you have occasionally network blips and never need to restart your MySQL boxes, these errors can accumulate over time and eventually cause you middle of the night pain.

See Host 'host name' is blocked in the MySQL docs. Sadly, there is no way to disable this check entirely. Setting the variable to 0 doesn't accomplish that. Your only real solutions are (a) setting it to a very high value (max_connect_errors=1844674407370954751), and (b) running an occasional FLUSH HOSTS command.


This is related to the above problem. In situations of network congestion (either at the client or server), it's possible for an initial connection to take several seconds to complete. But the default value for connect_timeout is 5 seconds. When you trip over that, the max_connect_errors problem above kicks in.

To avert this, try setting connect_timeout to a value more like 15 or 20. And also consider making thread_cache_size a non-zero value. That will help in situations when the server occasionally gets a high number of new connections in a very short period of time.


MySQL does a reverse DNS lookup on every incoming connection by default. This sucks. It seems that no matter how good your infrastructure is, there are blips in DNS service. MySQL's host cache exists to keep those lookups to a minimum. Yet I've seen this cause pain off and on for eight years now. I can only assume there's a bug in the host cache or the resolver library when this happens.

I recommend adding skip-name-resolve to your /etc/my.cnf to skip DNS entirely. Just use IP addresses or ranges for your GRANTs. It seems that slow replies from DNS servers can also help you to trip over connect_timeout as well. Imagine having 2 or 3 DNS servers configured but the first one is unavailable.


When the network connection between a master and slave database is interrupted in a way that neither side can detect (like a firewall or routing change), you must wait until slave_net_timeout seconds have passed before the salve realizes that something is wrong. It'll then try to reconnect to the master and pick up where it left off. That's awesome.

However, the default value is 3600 seconds. That's a full hour! FAIL.

Who wants their slaves to sit idle for that long before checking to see if something might be wrong? I can't think of anyone who wants that.

My suggestion, if you're in a busy environment, is that you set that to something closer to 30 seconds.