Subscribe: Eric.Weblog()
Added By: Feedage Forager Feedage Grade B rated
Language: English
android  app  blog entry  database  library  microsoft  mobile  mongo  people  server  sql server  sql  sqlite library  sqlite 
Rate this Feed
Rate this feedRate this feedRate this feedRate this feedRate this feed
Rate this feed 1 starRate this feed 2 starRate this feed 3 starRate this feed 4 starRate this feed 5 star

Comments (0)

Feed Details and Statistics Feed Statistics
Preview: Eric.Weblog()

Eric Sink

SourceGear Founder

Copyright: Copyright 2001-2015 Eric Sink. All Rights Reserved

SQLite and Android N

Wed, 15 Jun 2016 12:00:00 CST

TLDR The upcoming release of Android N is going to cause problems for many apps that use SQLite. In some cases, these problems include an increased risk of data corruption. History SQLite is an awesome and massively popular database library. It is used every day by billions of people. If you are keeping a list of the Top Ten Coolest Software Projects Ever, SQLite should be on the list. Many mobile apps use SQLite in one fashion or another. Maybe the developers of the app used the SQLite library directly. Or maybe they used another component or library that builds on SQLite. SQLite is a library, so the traditional way to use it is to just link it into your application. For example, on a platform like Windows Phone 8.1, the app developer simply bundles the SQLite library as part of their app. But iOS and Android have a SQLite library built-in to the platform. This is convenient, because developers do not need to bundle a SQLite library with their software. However The SQLite library that comes with Android is actually not intended to be used except through the android.database.sqlite Java classes. If you are accessing this library directly, you are actually breaking the rules. And the problem is Beginning with Android N, these rules are going to be enforced. If your app is using the system SQLite library without using the Java wrapper, it will not be compatible with Android N. Does your app have this problem? If your app is breaking the rules, you *probably* know it. But you might not. I suppose most Android developers use Java. Any app which is only using android.database.sqlite should be fine. But if you are using Xamarin, it is rather more likely that your app is breaking the rules. Many folks in the Xamarin community tend to assume that "SQLite is part of the platform, so you can just call it". Xamarin.Android 6.1 includes a fix for this problem for Mono.Data.Sqlite (see their release notes). However, that is not the only way of accessing SQLite in the .NET/Xamarin world. In fact, I daresay it is one of the less common ways. Perhaps the most popular SQLite wrapper is sqlite-net (GitHub). If you are using this library on Android and not taking the extra steps to bundle a SQLite library, your app will break on Android N. Are you using Akavache? Or Couchbase Lite? Both of these libraries use SQLite under the hood (by way of SQLitePCL.raw, which I maintain), so your app will need to be updated to work on Android N. There are probably dozens of other examples. GitHub says the sqlite-net library has 857 forks. Are you using one of those? Do you use the MvvmCross SQLite plugin? Do any of the components or libraries in your app make use of SQLite without you being aware of it? And the Xamarin community is obviously not the whole story. There are dozens of other ways to build mobile apps. I can think of PhoneGap/Cordova, Alpha Anywhere, Telerik NativeScript, and Corona, just off the top of my head. How many of these environments (or their surrounding ecosystems) provide (perhaps accidentally) a rule-breaking way to access the Android system SQLite? I don't know. What I *do* know is that even Java developers might have a problem. It's even worse than that Above, I said: "Any app which is only using android.database.sqlite should be fine." The key word here is "only". If you are using the Java classes but also have other code (perhaps some other library) that accesses the system SQLite, then you have the problems described above. But you also have another problem. To fix this, you are going to have to modify that "other code" to stop accessing the system SQLite library directly. One way to do this is to change the other code to call through android.database.sqlite. But that might be a lot of work. Or that other code might be a 3rd party library that you do not maintain. So you are probably interested in an easier solution. Why not just bundle another instance of the SQLite library into your app? This is what people who use sqlite-net on Xamarin will need to do, so it [...]

MongoDB and WTFs and Anger

Mon, 20 Jul 2015 12:00:00 CST

Recently, Sven Slootweg (joepie91) published a blog entry entitled Why you should never, ever, ever use MongoDB. It starts out with the words "MongoDB is evil" and proceeds to give a list of negative statements about same.

I am not here to respond to each of his statements. He labels them as "facts", and some (or perhaps all) of them surely are. In fact, for now, let's assume that everything he wrote is correct. My point here is not to say that the author is wrong.

Rather, my point here is that this kind of blog entry tells me very little about MongoDB while it tells me a great deal about the emotions of the person who wrote it.

Like I said, it may be true that every WTF the author listed is correct. It is also true that some software has more WTFs than others.

I'm not a MongoDB expert, but I've been digging into it quite a bit, and I could certainly make my own list of its WTFs. And I would also admit that my own exploration of Couchbase has yielded fewer of those moments. Therefore, every single person on the planet who chooses MongoDB instead of Couchbase is making a terrible mistake, right?

Let me briefly shift to a similar situation where I personally have a lot more knowledge: Microsoft SQL Server vs PostgreSQL. For me, it is hard to study SQL Server without several WTF moments. And while PostgreSQL is not perfect, I have found that a careful study there tends to produce more admiration than WTFs.

So, after I discovered that (for example) SQL Server has no support for deferred foreign keys, why didn't I write a blog entry entitled "Why you should never, ever, ever use SQL Server"?

Because I calmed down and looked at the bigger picture.

I think I could make an entirely correct list of negative things about SQL Server that is several pages long. And I suppose if I wanted to do that, and if I were really angry while I was writing it, I would include only the facts that support my feelings, omitting anything positive. For example, my rant blog entry would have no reason to acknowledge that SQL Server is the mostly widely used relational database server in the world. These kinds of facts merely distract people from my point.

But what would happen if I stopped writing my rant and spent some time thinking about the fact I just omitted?

I just convinced myself that this piece of software is truly horrible, and yet, millions of people are using it every day. How do I explain this?

If I tried to make a complete list of theories that might fit the facts, today's blog entry would get too long. Suffice it to say this: Some of those theories might support an anti-Microsoft rant (for example, maybe Microsot's field sales team is really good at swindling people), but I'm NOT going to be able to prove that every single person who chose SQL Server has made a horrible mistake. There is no way I can credibly claim that PostgreSQL is the better choice for every single company simply because I admire it. Even though I think (for example) that SQL Server handles NULL and UNIQUE in a broken way, there is some very large group of people for whom SQL Server is a valid and smart choice.

So why would I write a blog entry that essentially claims that all SQL Server users are stupid when that simply cannot be true? I wouldn't. Unless I was really angry.

MongoDB is undisputably the top NoSQL vendor. It is used by thousands of companies who serve millions of users every day. Like all young software serving a large user base, it has bugs and flaws, some of which are WTF-worthy. But it is steadily getting better. Any discussion of its technical deficiences which does not address these things is mostly just somebody venting emotion.


My initial experience with Rust

Mon, 08 Jun 2015 12:00:00 CST

First, a digression about superhero movies I am apparently incapable of hating any movie about a comic book superhero. I can usually distinguish the extremes. Yes, I can tell that "The Dark Knight" was much better than "Elektra". My problem is that I tend to think that the worst movies in this genre are still pretty good. And I have the same sort of unreasonable affection toward programming languages. I have always been fascinated by languages, compilers, and interpreters. My opinions about such things skew toward the positive simply because I find them so interesting. I do still have preferences. For example, I tend to like strongly typed languages more. In fact, I think it is roughly true that the stricter a compiler is, the more I like it. But I can easily find things to like in languages that I mostly dislike. I've spent more of my career writing C than any other language. But in use cases where I need something like C, I am increasingly eager for something more modern. I started learning Rust with two questions: How successful might Rust become as a viable replacement for C? If I enjoy functional programming, how much of that enjoyment can I retain while coding in Rust? The context My exploration of Rust has taken place in one of my side projects: LSM is a key-value database with a log-structured merge tree design. It is conceptually similar to Google LevelDB. I first wrote it in C#. Then I rewrote/ported it to F#. Now I have ported it to Rust. (The Rust port is not yet mentioned in the README for that repo, but it's in the top-level directory called 'rs'.) For the purpose of learning F# and Rust, my initial experience was the same. The first thing I did in each of these languages was to port LSM. In other words, the F# and Rust ports of LSM are on equal footing. Both of them were written by someone who was a newbie in the language. Anyway, although Rust and F# are very different languages, I have used F# as a reference point for my learning of Rust, so this blog entry walks that path as well. This is not to say that I think Rust and F# would typically be used for the same kinds of things. I can give you directions from Denver to Chicago without asserting they are similar. Nonetheless, given that Rust is mostly intended to be a modern replacement for C, it has a surprising number of things in common with F#. The big comparison table   F# Rust Machine model Managed, .NET CLR Native, LLVM Runtime CLR None Style Multi-paradigm, functional-first Multi-paradigm, imperative-first Syntax family ML-ish C-ish Blocks Significant whitespace Curly braces Exception handling Yes No Strings .NET (UTF-16) UTF-8 Free allocated memory Automatic, garbage collector Automatic, static analysis Type inference Yes, but not from method calls Yes, but only within functions Functional immutable collections Yes No Currying Yes No Partial application Yes No Compiler strictness Extremely strict Even stricter Tuples Yes Yes Discriminated unions type Blob = | Stream of Stream | Array of byte[] | Tombstone enum Blob { Stream(Box), Array(Box), Tombstone, } Mutability To be avoided Safe to use Lambda expressions let f = (fun acc item -> acc + item) let f = |acc, &item| acc + item; Higher-order functions List.fold f 0 a a.iter().fold(0, f) [...]

Announcing Zumero for SQL Server, Release 2.0

Fri, 08 May 2015 12:00:00 CST

Zumero for SQL Server (ZSS) is a solution for replication and sync between SQL Server and mobile devices. ZSS can be used to create offline-friendly mobile apps for iOS, Android, Windows Phone, PhoneGap, and Xamarin.

Our 2.0 release is a major step forward in the maturity of the product.


  • Compatibility with Azure SQL -- This release offers improved compatibility with Microsoft Azure SQL Database. Whether you prefer cloud or on-prem, ZSS 2.0 is a robust sync solution.

  • Improved filtering -- In the 2.0 release, filters have become more powerful and easier to use. Arcane limitations of the 1.x filtering feature have been lifted. New capabilities include filtering by date, and filtering of incoming writes.

  • Schema change detection -- The handling of schema changes is now friendlier to the use of other SQL tools. In 1.x, schema changes needed to be performed in the ZSS Manager application. In 2.0, we detect and handle most schema changes automatically, allowing you to integrate ZSS without changing the way you do things.

  • Better UI for configuration -- The ZSS Manager application has been improved to include UI for configuration of multiple backend databases, as well as more fine-grained control of which columns in a table are published for sync.

  • Increased Performance -- Perhaps most important of all, ZSS 2.0 is faster. In some situations, it is a LOT faster.

Lots more info at


Microsoft is becoming cool again

Thu, 30 Apr 2015 12:00:00 CST

Isn't "New Microsoft" awesome?

.NET is going open source? And cross-platform? On Github?!?

The news out of Redmond often seems like a mis-timed April fools joke.

But it's real. This is happening. Microsoft is apparently willing to do "whatever it takes" to get developers to love them again.

How did this company change so much, so quickly?

A lot of folks are giving credit to CEO Satya Nadella. And there could be some truth to that. Maaaaaaybe.

Another popular view: Two of the most visible people in this story are:

  • Scott Hanselman (whose last name I cannot type without double-checking the spelling.)

  • and Scott Gu (whose last name is Guenther. Or something like that. I can never remember anything after the first two letters.)

I understand why people think maybe these two guys caused this revolution. They both seem to do a decent job I suppose.

But the truth is that New Microsoft started when Microsoft hired Martin Woodward.

What?!? Who the heck is Martin Woodward?

Martin's current position is Executive Director of the .NET Foundation. Prior to that, he worked as a Program Manager on various developer tools.

Nobody knows who Martin is. Either of the Scotts have 2 orders of magnitude more Twitter followers.

But I think if you look closely at Martin's five+ year career at Microsoft, you will see a pattern. Every time a piece of Old Microsoft got destroyed in favor of New Microsoft, Martin was nearby.

Don't believe me? Ask anybody in DevDiv how TFS ended up getting support for Git.

It's all about Martin.

So all the credit should go to Martin Woodward then?

Certainly not.

You see, Martin joined Microsoft in late 2009 as part of their acquisition of Teamprise.

And Teamprise was a division of SourceGear.

I hired Martin Woodward (single-handedly, with no help or input from anybody else) in 2005. Four years later, when Microsoft acquired our Teamprise division (which I made happen, all by myself), Martin became a Microsoft employee.

Bottom line: Are you excited about all of the fantastic news coming out of Build 2015 this week? That stuff would never have happened if it were not for ME.

Eric, how can we ever thank you?

So, now that you know that I am the one behind all the terrific things Microsoft is doing, I'm sure you want to express your appreciation. But that won't be necessary. While I understand the sentiment, in lieu of expensive gifts and extravagant favors, I am asking my adoring fans to do two things:

First, try not to forget all the little people at Microsoft who are directly involved in the implementation of change. People like Martin, or the Scotts, or Satya. Even though these folks are making a relatively minor contribution compared to my own, I would like them to be supported in their efforts. Be positive. Don't second-guess their motives. Lay aside any prejudices you might have from Old Microsoft. Believe.

Second, get involved. Interact with New Microsoft as part of the community. Go to Github and grab the code. Report an issue. Send a pull request.

Embellishments and revisionist history aside...

Enjoy this! It's a GREAT time to be a developer.


What Mongo-ish API would mobile developers want?

Mon, 27 Apr 2015 12:00:00 CST

A couple weeks ago I blogged about mobile sync for MongoDB. Updated Status of Elmo Embeddable Lite Mongo continues to move forward nicely: Progress on indexes: Compound and multikey indexes are supported. Sparse indexes are not done yet. Index key encoding is different from the KeyString stuff that Mongo itself does. For encoding numerics, I did an ugly-but-workable F# port of the encoding used by SQLite4. Hint is supported, but is poorly tested so far. Explain is supported, partially, and only for version 3 of the wire protocol. More work to do there. The query planner (which has delusions of grandeur for even referring to itself by that term) isn't very smart. Indexes cannot yet be used for sorting. Indexes are currently never used to cover a query. When grabbing index bounds from the query, $elemMatch is ignored. Because of this, and because of the way Mongo multikey indexes work, most index scans are bounded at only one end. The $min and $max query modifiers are supported. The query planner doesn't know how to deal with $or at all. Progress on full-text search: This feature is working for some very basic cases. Phrase search is not implemented yet. Language is currently ignored. The matcher step for $text is not implemented yet at all. Everything within the index bounds will get returned. The tokenizer is nothing more than string.split. No stemming. No stop words. Negations are not implemented yet. Weights are stored in the index entries, but textScore is not calculated yet. I also refactored to get better separation between the CRUD logic and the storage of bson blobs and indexes (making it easier to plug-in different storage layers). Questions about client-side APIs So, let's assume you are building a mobile app which communicates with your Mongo server in the cloud using a "replicate and sync" approach. In other words, your app is not doing its CRUD operations by making networking/REST calls back to the server. Instead, your app is working directly with a partial clone of the Mongo database that is right there on the mobile device. (And periodically, that partial clone is magically synchronized with the main database on the server.) What should the API for that "embedded lite mongo" look like? Obviously, for each development environment, the form of the API should be designed to feel natural or native in that environment. This is the approach taken by Mongo's own client drivers. In fact, as far as I can tell, these drivers don't even share much (or any?) code. For example, the drivers for C# and Java and Ruby are all different, and (unless I'm mistaken) none of them are mere wrappers around something lower level like the C driver. Each one is built and maintained to provide the most pleasant experience to developers in a specific ecosystem. My knee-jerk reaction here is to say that mobile developers might want the exact same API as presented by their nearest driver. For example, if I am building a mobile app in C# (using the Xamarin tools), there is a good chance my previous Mongo experience is also in C#, so I am familiar with the C# driver, so that's the API I want. Intuitive as this sounds, it may not be true. Continuing with the C# example, that driver is quite large. Is its size appropriate for use on a mobile device? Is it even compatible with iOS, which requires AOT compilation? (FWIW, I tried compiling this driver as a PCL (Portable Class Library), and it didn't Just Work.) For Android, the same kinds of questions would need to be asked about the Mongo Java driver. And then there are Objective-C and Swift (the primary developer platform for iOS), for whic[...]

Mobile Sync for Mongo

Mon, 13 Apr 2015 12:00:00 CST

We here at Zumero have been exploring the possibility of a mobile sync solution for MongoDB. We first released our Zumero for SQL Server product almost 18 months ago, and today there are bunches of people using mobile apps which sync using our solution. But not everyone uses SQL Server, so we often wonder what other database backends we should consider supporting. In this blog entry, I want to talk about some progress we've made toward a "Zumero for Mongo" solution and "think out loud" about the possibilities. Background: Mobile Sync The basic idea of mobile sync is to keep a partial copy of the database on the mobile device so the app doesn't have to go back to the network for every single CRUD operation. The benefit is an app that is faster, more reliable, and works offline. The flip side of that coin is the need to keep the mobile copy of the database synchronized with the data on the server. Sync is tricky, but as mobile continues its explosive growth, this approach is gaining momentum: Legendary developer Dan Bricklin: Forrester analyst Michael Facemire: If the folks at Mongo are already working on something in this area, we haven't seen any sign of it. So we decided to investigate some ideas. Pieces of the puzzle In addition to the main database (like SQL Server or MongoDB or whatever), a mobile sync solution has three basic components: Mobile database Runs on the mobile device as part of the app Probably an embedded database library Keeps a partial replica of the main database Wants to be as similar as possible to the main database Sync server Monitors changes made by others to the main database Sends incremental changes back and forth between clients and the main database Resolves conflicts, such as when two participants want to change the same data Manages authentication and permissions for mobile clients Filters data so that each client only gets what it needs Sync client Monitors changes made by the app to the mobile database Talks over the network to the sync server Pushes and pulls incremental changes to keep the mobile database synchronized For this blog entry, I want to talk mostly about the mobile database. In our Zumero for SQL Server solution, this role is played by SQLite. There are certainly differences between SQL Server and SQLite, but on the whole, SQLite does a pretty good job pretending to be SQL Server. What embedded database could play this role for Mongo? This question has no clear answer, so we've been building a a lightweight Mongo-compatible database. Right now it's just a prototype, but its development serves the purpose of helping us explore mobile sync for Mongo. Embeddable Lite Mongo Or "Elmo", for short. Elmo is a database that is designed to be as Mongo-compatible as it can be within the constraints of mobile devices. In terms of the status of our efforts, let me begin with stuff that does NOT work: Sharding is an example of a Mongo feature that Elmo does not support and probably never will. Elmo also has no plans to support any feature which requires embedding a JavaScript engine, since that would violate Apple's rules for the App Store. We do hope to support full text search ($text, $meta, etc), but this is not yet implemented. Similarly, we have not yet implemented any of the geo features, but we consider them to be within the scope of the project. Elmo does not support capped collections, and we are not yet sure if it should. [...]

Improvements in Xamarin.Forms 1.3

Tue, 27 Jan 2015 12:00:00 CST

Back in November I wrote a blog entry about performance problems resulting from the design of the layout system in Xamarin.Forms. I am pleased to report that things took a big step forward with the recent release of version 1.3. Reviewing the problem In a nutshell, the Layout classes do too much. They contain functionality to make sure everything gets updated whenever something changes. In principle, this is good, since we obviously don't want stale stuff on the screen. But in practice, there are many cases where the built-in update code ends up being slower than necessary. For example, suppose I'm going to add ten child views to a layout. With the built-in update code, a layout cycle will get triggered ten times, once for each child view I add. Worse, if I'm trying to do any kind of subview recycling, the odds are high that I want to add a child view while I am processing a layout cycle. This will trigger a recursive layout cycle, resulting in the end of civilization as we know it. Instead, what I want is one layout cycle which happens after all ten child views have been added. The solution I proposed IMHO, the best design for this kind of problem is to have multiple layers: The Low-Level layer models child view relationships only. It provides a way for a View to be inside another View, but it doesn't give much more than that. In iOS terms, this is UIView.addSubView. The High-Level layer (which is built on the functionality provided by the layers below it) has Views which actively manage their child views. In iOS terms, an example of this would be UICollectionView. In the Middle, it would make sense to have a layer which provides things which are common to all (or nearly all) of the stuff in the High-Level layer, to avoid code duplication. Xamarin.Forms has the High-Level layer and the Middle layer, but it does not have the Low-Level layer. So I proposed creating it. I didn't get exactly what I wanted, but... The solution in Xamarin.Forms 1.3 In Xamarin.Forms 1.3, the Middle layer is still the lowest thing we've got. However, there are new capabilities which allow the Middle layer to pretend like it is a Low-Level layer. It still has a bunch of built-in update code, but now that code can be turned off. :-) The important new capabilities are: ShouldInvalidateOnChildAdded ShouldInvalidateOnChildRemoved OnChildMeasureInvalidated By returning false from my override of ShouldInvalidateOnChildAdded() and ShouldInvalidateOnChildRemoved(), I can have a Layout which doesn't do any automatic updates when I add or remove children. And by overriding OnChildMeasureInvalidated(), I can have a Layout which refuses to do real estate negotiations with its child views. This is good. How I'm using this Because of this new stuff, an upcoming release of our DataGrid component will be even faster. Our panel layout class will look something like this: private class myLayout : Layout { Func getBox; public myLayout(Func f) { getBox = f; } public void LayoutOneChild(View v) { Rectangle r = getBox (v); v.Layout (r); } public void LayoutAllChildren() { foreach (View v in Children) { LayoutOneChild (v); } } protected override bool ShouldInvalidateOnChildAdded (View child) { return false; // stop pestering me } protected override bool ShouldInvalidateOnChildRemoved (View child) { return false; // go away and leave me alone } protected override void OnChildMeasureInvalidated () { // I'm ignoring you. You'll take whatever size I want to give // you. And you'll like it. } protected override void LayoutChildren (double x, double y, double width, double [...]

Why your F# evangelism isn't working

Mon, 05 Jan 2015 12:00:00 CST

Ouch. Eric, you're one of those anti-F# people, aren't you? Nope. If you skim this blog entry too quickly or just read the title, you might think I am someone who does not like F#. Nothing could be further from the truth. Over the last several months, I have become a big F# fan. It has become my preferred language for personal projects. My current side project is a key-value store in F#. I have learned a lot by writing it, and I am even starting to think it might end up becoming useful. :-) Mostly, I find coding in F# to be extremely satisfying. I am writing this article not as an opponent of F#, but rather, as someone who hopes that F# will become a mainstream .NET language. Eric, you're wrong. F# is mainstream already. Of course it is. For some definition of "mainstream". F# is gaining traction really fast. People are using F# for real stuff. The language is improving. Xamarin is supporting it. By nearly any measure, F# is showing a lot of momentum over the last few years. If you are an F# fan, there just isn't any bad news running around. But for this article, I am using a definition of "mainstream", (which I'll explain below) which I believe F# has not yet reached. If, when you arrive at the end of this article, you do not like my definition of mainstream, that's okay. Just take a copy of this article, and do a search and replace all instances of the word "mainstream" with "purple". I have no desire to argue with you about what "mainstream" means, but if you want to argue about the meaning of "purple", I'll be happy to. :-) You're wrong again. My F# evangelism IS working Of course it is. To a certain extent. But in marketing terminology, as far as I can tell, most F# users today are "early adopters". Very few are "pragmatists". And F# has not yet "crossed the chasm". What is "the chasm"? The term originates from a 1991 book by Geoffrey Moore. The main point of Moore's book is that the classical marketing bell curve has a problem. Typically (and, prior to Moore's book, always), that bell curve is drawn like this: The basic idea of this curve is that when a market adopts a new techology, it follows a pattern. The technology moves from left to right on the bell curve, becoming adopted by four groups in the following order: the "early adopters" (people who like trying new technologies) the "pragmatists" (people who only care about technology to get something done) the "conservatives" (pragmatists, but even more risk-averse) the "laggards" (people who actively avoid new things) Together, the pragmatists and conservatives are the definition of "mainstream" for the purpose of this article. Moore's key observation is that moving from the early adopters to the pragmatists is very hard. Some technologies never make it. To illustrate this problem, Moore suggests drawing the bell curve differently, with a "chasm" between the early adopters and the pragmatists: His book explains why the chasm exists, why some technologies "die at the bottom of the chasm", and how some technologies successfully "cross the chasm". It is a marketing classic and highly recommended. For the purpose of this blog entry, the main thing you need to know is this: The chasm exists because pragmatists generally adopt new techologies as a herd. They don't adopt a new technology until they see other pragmatists using it. This creates a chicken-and-egg problem. How does this herd thing work? Pragmatists have an annual conference where they all agree to stay with their existing technologies. The actual vote is not formal, but consensus gets reached. A lot of this process happens in hallways and the dining hall: "Are you considering Windows 8 or should we all just stay with Windows 7 and see what happens next?" Some of the process happens in the conference[...]

Do elite software developers exist?

Sat, 03 Jan 2015 12:00:00 CST

People have been debating for decades about the notion of a software developer who is "elite". Sometimes this person is described as a "rockstar developer" or a "10X" developer. Not everyone agrees about this issue. A little over a year ago, Scott Hanselman wrote a blog entry entitled The Myth of the Rockstar Programmer, which makes some very good points and mostly argues against the notion of a developer who is dramatically more talented than average. More recently Paul Graham wrote an essay entitled Let the Other 95% of Great Programmers In, which asserts that some programmers are exceptional and proceeds from that notion to an argument for change in immigration policy. I want to shine a light on this question by comparing the software development profession to a couple of others. Sports In the world of sports, some people are simply a LOT more talented than others. I know of basketball player who was amazing in high school. He was the most valuable player at his school, and he led his team to many victories. But when he tried to make the team at a junior college, he just wasn't good enough. And most juco players probably would have no chance at making the roster of a mid-major school like Bradley. And most of the players on Bradley's roster probably wouldn't make it at a major conference school like Nebraska. And in any given year, the players at Nebraska probably wouldn't be good enough to start at an elite school like Kentucky. And most of the players on Kentucky rosters just aren't good enough to play in the NBA. And lots of guys make the NBA but just aren't good enough to be a starter. And the average starter in the NBA is far less talented than Michael Jordan or LeBron James. I could cite lots of other examples from golf, tennis, American football, and so on. All sports have very large distances between good and elite. And in general, that distance is composed mostly of talent, not preparation. A mid-level PGA tour golfer can get better by practice and training, but such disciplines will never help him reach the level of Tiger Woods or Jack Nicklaus. In sports, the existence of elite participants is widely understood and accepted. In part, this is because sports have good ways of keeping score. They have commonly accepted ways to measure things. At the bare minimum, for each contest, we need a way to know who won and who lost. It is generally understood that a single contest is not always won by the most talented player or team. If the stronger player has a bad day and the weaker player has a good day, you can get an "upset". But if your overall record after some number of contests is sufficiently strong, you will be considered more talented, because no other explanation fits the data. If your record against Roger Federer is 1-0, nobody is sure what to think about you. You might be more talented than Roger. More than likely, you are not. But it's really hard to claim you have more talent than Roger Federer if your record against him is 1-20. The scoring system is what enables us to wonder if Roger Federer might be the greatest tennis player of all time. Suppose I woke up tomorrow morning with the ability to always hit a golf ball into the cup from 150 yards out. This feat is obviously possible, since I did it once. Now let's pretend that I could do it every time. Because of the scoring system, that's all I would need. I don't need luck. I don't need to know "the right people". I simply have to play the US open qualifying rounds, (which anybody can do) and I win them all. Then I win the US open. That gets me invited to the other tournaments. And I win them all. Because of talent and a scoring system, Sports Illustrated would be forced to run a cover photo of the best golfer in the world, an overweight middle-aged gu[...]