Subscribe: George V. Reilly's Technical Blog
http://weblogs.asp.net/george_v_reilly/rss.aspx
Added By: Feedage Forager Feedage Grade B rated
Language: English
Tags:
application  bit  book  card  files  git  iis  javascript  much  new  photos  programfiles  python  regex  server  win  windows  work 
Rate this Feed
Rate this feedRate this feedRate this feedRate this feedRate this feed
Rate this feed 1 starRate this feed 2 starRate this feed 3 starRate this feed 4 starRate this feed 5 star

Comments (0)

Feed Details and Statistics Feed Statistics
Preview: George V. Reilly's Technical Blog

George V. Reilly's Technical Blog





 



Sample ASP Components: now at Github

Thu, 12 Dec 2013 19:23:00 GMT

(image)

From October 1996 to May 1997, I wrote a number of sample components for the then-new Active Server Pages (Classic ASP). I worked for MicroCrafts, a consulting company in Redmond, WA; the samples were written for Microsoft's Internet Information Server (IIS) team. Most of the components used Microsoft's new Active Template Library (ATL), a C++ library for COM.

This work had two important consequences for me: Microsoft recruited me to join the IIS development team to work on improving ASP performance for IIS 3, and Wrox Press invited me to write Beginning ATL COM Programming I was originally supposed to be the sole author of the book, but I was a slow writer and I was caught up in the IIS 4 deathmarch, so Wrox brought in three more co-authors to complete the book. A fourth co-author was brought in for the second edition, Beginning ATL 3 COM Programming. As for IIS, I spent seven years on the team, where in addition to leading the performance team, I also worked on the http.sys kernel driver that was released in Windows Server 2003 (IIS 6).

For many years, these components could be found at http://www.georgevreilly.com/sample-ASP-components.html. I'm making them available now at Github.




Turn off Windows Defender on your builds

Mon, 11 Nov 2013 08:38:00 GMT

I've spent some time this evening profiling a Python application on Windows, trying to find out why it was so much slower than on Mac or Linux. The application is an in-house build tool which reads a number of config files, then writes some output files.

Using the RunSnakeRun Python profile viewer on Windows, two things immediately leapt out at me: we were running os.stat a lot and file.close was really expensive.

A quick test convinced me that we were stat-ing the same files over and over. It was a combination of explicit checks and implicit code, like os.walk calling os.path.isdir. I wrote a little cache that memoizes the results, which brought the cost of the os.stats down from 1.5 seconds to 0.6.

Figuring out why closing files was so expensive was harder. I was writing 77 files, totaling just over 1MB, and it was taking 3.5 seconds. It turned out that it wasn't the UTF-8 codec or newline translation. It was simply that closing those files took far longer than it should have.

I decided to try a different profiler, hoping to learn more. I downloaded the Windows Performance Toolkit. I recorded a couple of traces of my application running, then I looked at them in the Windows Performance Analyzer, whereupon I saw that in each case, the CPU spike of my app was followed by a CPU spike in MsMpEng.exe.

What's MsMpEng.exe? It's Microsoft's antimalware engine, at the heart of Windows Defender. I added my build tree to the list of excluded locations, and my runtime halved. The 3.5 seconds of file closing dropped to 60 milliseconds, a 98% reduction.

The moral of this story is: don't let your virus checker run on your builds.

(image)



Review: Backbone Testing

Thu, 24 Oct 2013 07:12:00 GMT

Title: Backbone Testing Author: Ryan Roemer Rating: $stars(4.5) Publisher: Packt Copyright: 2013 ISBN: 178216524X Pages: 168 Keywords: programming, testing, javascript, backbone, mocha, chai, sinon Reading period: October 2013 Backbone Testing is a short, dense introduction to testing JavaScript applications with three testing libraries, Mocha, Chai, and Sinon.JS. Although the author uses a sample application of a personal note manager written with Backbone throughout the book, much of the material would apply to any JavaScript client or server framework. Mocha is a test framework that can be executed in the browser or by Node, which runs your tests. Chai is a framework-agnostic TDD/BDD assertion library. Sinon.JS provides standalone test spies, stubs and mocks for JavaScript. They complement each other and the author does a good job of explaining when and how to use each. I've written a lot of tests in Python (unittest and mock, primarily) and C# (NUnit), but my experience with JavaScript unit testing was both limited and years out of date. The JavaScript ecosystem continues to evolve rapidly, with new browser frameworks and Node packages springing up everywhere. JavaScript has some particular challenges in testing—notably, the DOM, asynchrony, and callbacks. Mocha, Chai, and Sinon meet those challenges, though they can't take away all the pain. The author describes how to test Backbone models, views, and collections; dealing with asynchrony; provides useful testing heuristics, including isolating components to reduce dependencies; when to use stubs and mocks and fake servers; and test automation with PhantomJS. He does not, however, teach you Backbone itself; for that, you'll need another book. There are a few areas which I thought were dealt with too lightly. There's no real discussion of Test-driven_development or Behavior-driven_development, which provide the intellectual foundations of much of the book. Nor does he have much to say about testability and how to make legacy code more testable. The sample Notes app has plenty of testing seams (much of this falls naturally out of the architecture of Backbone); other people's apps are not so lucky. The chapter on automation is extremely terse—it could be expanded into a very large book!—but it does provide useful indicators to many areas for exploration. I learned a lot from this book and I have no hesitation in recommending it. Disclosure: Thanks to Ryan Roemer and Packt for a review copy of this book. [...]



Building a static website

Sun, 20 Jan 2013 08:11:00 GMT

I've spent time over the last three weeks working on a new website for the Northwest C++ Users' Group. I blogged about the NWCPP website refresh over there. In brief, I moved the website from an instance of the Joomla Content Management System at Just Host to a static website generated by Pelican and hosted at Github Pages, and I'm happy with the results.




SerializationException: the constructor was not found

Fri, 22 Apr 2011 22:39:00 GMT

(image)

I just finished another post at the Cozi Tech Blog, SerializationException: the constructor was not found




Security 101 for Developers

Fri, 22 Apr 2011 22:31:00 GMT

(image)

The Cozi Tech Blog needed some love, so I wrote a post a couple of weeks ago on Security 101 for Developers.




Generating UUIDs in JavaScript

Tue, 04 May 2010 22:15:00 GMT

(image)

Another Cozi Tech Blog post: Generating UUIDs in JavaScript




SQLAlchemy Sharding

Sun, 04 Apr 2010 07:21:00 GMT

(image)

I wrote up some lessons that I learned about SQLAlchemy Sharding at the Cozi Tech Blog.




Engineering Jobs at Cozi

Sat, 27 Mar 2010 06:03:00 GMT

Cozi is hiring. We have positions in Web Development, Software Engineering, and System Engineering at our headquarters in Seattle.

Full details at the Careers Page.




StackOverflow DevDays Seattle 2009

Tue, 27 Oct 2009 07:17:00 GMT

I spent last Wednesday at Benaroya Hall, attending the Seattle edition of StackOverflow's traveling DevDays conference. It was well worth $99. Joel Spolsky, owner of FogCreek Software and co-founder of StackOverflow, opened the conference with a keynote about the dichotomy of power and simplicity. People are happier when not overwhelmed with choices. Many of the choices that software forces users to make are essentially meaningless to the users. However, even though people want simplicity, they also want features and different people use different features. Powerful software sells more copies. He argues that developers and designers should put in the extra work to make good choices on behalf of the users: don't make users feel bad about themselves. Undo is better than a confirmation dialog. You are not in charge of what your users do. Scott Hanselman spoke about ASP.NET MVC. We're moving away from ASP.NET to Python, but if we were to use ASP.NET again, MVC would be a compelling feature. His presentation was entertaining, if gimmicky. Rory Blyth introduced iPhone development, in a tone of snarky ambivalence. He mentioned the Stockholm Syndrome. He stressed that Apple's Design "Guidelines" are effectively laws: violate them and you won't make it into the App Store. Looks like there's a lot of tedious messing around to hook things up in Objective-C. At the very end, he briefly demoed MonoTouch, which seemeed a little less tedious. Cody Lindley introduced jQuery. I've done a lot of work with jQuery, but I still learned a few things. He worked through five facets of jQuery: Find something, do something; Create something, do something; Chaining; Implicit iteration; and jQuery parameters. He has an ebook at jqueryenlightenment.com, which I just picked up. Daniel Rocha of Nokia talked about the cross-platform Qt (/cute/) toolkit, which runs on Windows, Mac, and Linux. More importantly from Nokia's point of view, it runs on their smartphones. Nokia has changed the licensing of Qt—once very expensive for closed-source apps, it's now free for apps that don't modify the Qt source. Qt is for C++, but there are bindings for other languages, such as Python. Joel Spolsky came back and treated us to a half-hour demonstration of Fogbugz 7, Evidence-Based Scheduling, and Kiln, their new hosted Mercurial repository. Not terribly interesting to me, but the conference was only $99. Ted Leung gave us a rather dry Hacker's Introduction to Python from slides rendered unreadable by a poor choice of colors. I've done a lot of Python, so I didn't learn much new. pip is an easy_install replacement that uninstalls; zc.buildout assembles apps from multiple parts; bpython is a fancy REPL. Dan Sanderson talked about Google App Engine and demoed building apps with Java and with Python. Looked pretty cool and straightforward. We probably won't go that route, since we're pushing data to Amazon's S3, so EC2 makes more sense for us. Finally, Steve Seitz from the University of Washington gave a cool talk on Modeling the World from Internet Photos. Some of this technology ended up in Photosynth. See Building Rome in a Day for some demos. [...]



Review: Pragmatic Version Control Using Git

Mon, 19 Oct 2009 05:44:00 GMT

(image)
Title: Pragmatic Version Control Using Git
Author: Travis Swicegood
Rating: $stars(4)
Publisher: Pragmatic Bookshelf
Copyright: 2008
Pages: 179
Keywords: computers
Reading period: 10–18 October, 2009

As part of my personal conversion to Git, I read Swicegood's Git book. It's a decent introduction to Git and you learn how to do all the basic tasks as well as some more advanced topics. The examples are clear and well-paced.

I would have liked to see more about collaboration and workflow in a DVCS world, perhaps a few case studies: how is Git used in the Linux kernel development process; how a small, distributed team uses Git and GitHub; how a collocated team migrates from more traditional tools.

The book avoids discussing the lower levels of the Git object model, which is a reasonable choice for a pragmatic guide.

Recommended.




Third-Party Cookies

Sat, 17 Oct 2009 07:15:00 GMT

(image)

Over the last few weeks, I built a PHP application that overlays Approve 71 banners on profile pictures. The actual application is hosted in an iframe and lives on a server in a different domain, eq.dm, than the main server at approvereferendum71.org.

This works fine in most browsers. Then we started getting reports that it wasn't working in IE8 on Win7 RC1. The iframe content was blank.

Poking around, I found the problem with the Fiddler proxy. The landing page on eq.dm was supposed to stick some information into the PHP session, then redirect to a second page at the same site. The second page was in an endless loop, redirecting to itself. In Fiddler, I saw a different PHPSESSID cookie on each response, and no cookie in the requests.

After reading IE 8 only has access to session cookies, I told IE8 to Accept All Cookies and the iframe content appeared. That fixed it for me, but we could hardly ask people to lower their security sessions.

I created a P3P file for the second domain, using the IBM P3P Policy Editor. (KB 323752 has more background on P3P and third-party cookies.)

IE now worked at its default security level. Problem solved! Or so I thought.

A day later, we got reports of similar problems with Safari 4 on Mac OS X.

I sniffed the traffic with Wireshark. Same problem: the “third-party“ cookie wasn't being accepted by Safari.

Unfortunately, Setting cross-domain cookies in Safari indicated that there was no reasonable workaround.

We overcame the issue up playing some DNS games, which was only possible because we control both servers. The second server is now also acting as a subdomain of the first, at dev.approvereferendum71.org. We used ini_set("session.cookie_domain",".approvereferendum71.org") to scope the iframe cookies. I've tried it in a variety of Windows, Mac, and Linux browsers, and it works in all of them.




Gitting Along

Thu, 15 Oct 2009 15:24:00 GMT

(image)

In the last few weeks, I've switched over to Git for most of my version-control needs, at home and at work, after putting it on the long finger for months.

We continue to use Subversion at work, but I've recently followed Pavel and Eric's lead in using git-svn. I work locally on my own private branches and git svn dcommit and git svn rebase occasionally. I'm primarily on Windows at work, but I have a Linux box and a Mac Mini too, while at home, I have a MacBook, a Linux netbook, and a Vista desktop. I'm using msysGit, occasionally supplemented by TortoiseGit and QGit. Pavel's on a Mac and Eric's mostly on Ubuntu, so git adoption was easy for them.

When I first tried git-svn under msysGit about a year ago, it didn't work worth a damn. Git-svn works fine now, but it's slow compared to the *nix implementation. The developers say that's due to the fork() emulation of the MSys/Cygwin layer. The rest of msysGit is much faster.

For my home needs, I've had private Subversion repositories at DevjaVu.com and OpenSvn.csie.org. DevjaVu has gone out of business and OpenSvn has been unavailable too often for my liking. It was time to find some new hosting.

I've experimented with private Git repositories at GitHub and ProjectLocker. GitHub is very nice, but charges for private repositories. ProjectLocker provides free private repositories, but is comparatively clunky.

ProjectLocker lets you set up a fresh repository on their server. They tell you how to clone from that, which is great for a new repository. But they don't tell you how to hook it up to an existing local repository. Since I had some difficulty in figuring it out, here's the recipe:

git remote add origin git-foobar@freeN.projectlocker.com:foobar.git
git pull origin master
... merge, local edits and commits ...
git push origin master

I found Git, Xcode and ProjectLocker and Cygwin, SSH and ProjectLocker useful in figuring this out.




Launching 32-bit applications from batchfiles on Win64

Sat, 12 Sep 2009 06:16:00 GMT

(image)

I've been running the 64-bit version of Windows 7 RC since June. It's been quite painless on the whole.

One wrinkle that I ran into was with some batchfiles which launch applications in %ProgramFiles% (normally C:\Program Files). Due to the magic WOW64 redirector, 32-bit applications are actually installed into %ProgramFiles(x86)%—normally C:\Program Files (x86)—instead of %ProgramFiles%. This is transparent to the 32-bit applications, which think they're running in %ProgramFiles% (C:\Program Files).

However, the cmd.exe shell is 64-bit (unless you make a special effort to run the 32-bit cmd.exe in SysWOW64), so batchfiles see the 64-bit %ProgramFiles% which contains 64-bit applications.

Hence, a batchfile that launches an installed 32-bit application on Win64 must use %ProgramFiles(x86)%, not %ProgramFiles%.

It sounds trivial to have a batchfile detect which flavor of %ProgramFiles% it should use, but the parentheses in the environment variable name make it tricky to parse. On earlier versions of Win64, the environment variable was called %ProgramFilesx86%. Presumably they added the strange parentheses into the variable name because the directory name always included them.

Here's a tiny batchfile that will launch the 32-bit DiffMerge correctly on both Win64 and Win32 platforms.

@setlocal
@set _pf=%ProgramFiles%
@if not "[%ProgramFiles(x86)%]"=="[]" set _pf=%ProgramFiles(x86)%
@start "" /b "%_pf%\SourceGear\DiffMerge\DiffMerge.exe" %*

I long ago found that the safest way to test environment variables whose values may include spaces, is to surround them with both double quotes and square brackets.




Recovering photos from a corrupted card

Sat, 05 Sep 2009 07:45:00 GMT

(image)

I had about 60 apparently corrupted photos on a CompactFlash card the other evening. It might have been due to Lightroom going berserk, but it was more likely from my pulling the card reader out of the computer without ejecting it first.

The photos wouldn't show up under Mac, Linux, or Windows. I tried to chkdsk the card under Windows, which complained about a “raw” disk. That led me to ZAR, the Zero Assumption Recovery toolkit. The evaluation copy retrieved the photos very nicely. Whew!




XKCD's Tech Support Cheat Sheet

Tue, 25 Aug 2009 06:57:00 GMT

(image)

To my tech supportees:

XKCD's Tech Support Cheat Sheet nails it! This is essentially what I'm doing when I sit down at your computer and dig you out of your latest hole.

If only you'd make some more intelligent guesses for yourself, you might be able to solve more of your problems.

But it's not quite that simple.

You probably don't understand very much about what you're doing. I have enormous depth and breadth of experience which informs my investigation. I am—no false modesty here—a master of debugging. The extra context helps me hone in on a solution more often than not.

Still, do give the flowchart a shot. Being able to solve your own problems is very empowering.




Moving Photos in Lightroom

Tue, 18 Aug 2009 05:42:00 GMT

(image)

When using Lightroom before, I was never able to figure out how to move photos from one folder to another. You'd think that you could just click on a photo and drag it. I just spent twenty minutes figuring out what I've been doing wrong. After you've selected multiple photos, click on the photo thumbnail and not the surrounding gray frame, and then you can drag the photos to the target folder.

I had become accustomed to clicking on the frames to multi-select photos, so naturally I assumed that was also how you dragged a set of photos. But clicking on a frame of a selected photo merely deselects all the other photos.




64-bit Windows 7

Sat, 11 Jul 2009 06:12:00 GMT

I mentioned three weeks ago that I had just repaved my work dev box and installed the 64-bit version of the Windows 7 RC. Nine or ten years after I first ported parts of IIS to Win64, I am finally running my main desktop on 64-bit Windows. With one exception, it's been painless. Programs have just worked, devices have just worked. There are relatively few native x64 applications, but for the most part it doesn't matter. The cases where it does matter—e.g., shell extensions such as TortoiseSVN—are available as 64-bit binaries. I briefly flirted with using the 64-bit build of Python, but realized that I would have to recompile several eggs as 64-bit binaries. That was too painful and the 32-bit binary did everything I needed. Building in Visual Studio 2005 is noticeably faster. I'm not sure how much of it was due to accumulated cruft after 18 months on Vista, but builds there were very slow. The one exception was a major problem for the first week and a half. Whenever I ran our ASP.NET web application, it would go berserk, eat up all 4GB of my physical RAM, push the working set of IIS's w3wp.exe to 12GB, and max out one of my 4 cores! The only way to maintain any sanity was to run iisreset every 20 minutes to gently kill the process. WinDbg and Process Explorer showed that the rogue thread was stuck in a loop in mscorjit!LifetimesListInteriorBlocksHelperIterative. I passed a minidump on to my former colleagues in IIS, who sent it to the CLR team. They said: The only thing I can tell is that it is Regex, and some regex expression compiled down to a method with 456KB of IL. That is huge, and yes 12GB of RAM consumed for something like that is expected. With that clue, I was able to track down the problem, a particularly foul regex, built from a 10KB string, with 32 alternating expressions, each of which contains dozens of alternated subexpressions. The string is built from many smaller strings, so it's not obvious in the source just how ugly it is. I commented out the new Regex() and my problems went away. Regardless of how ugly the regex is, this is a major regression in the CLR. This code has been working without blatant problems for two years on the 32-bit flavors of XP, Server 2003, Vista, and Server 2008. I've been meaning to try this code on 32-bit Windows 7, but have been too busy. (The original, long-gone author was apparently aware that the regex is expensive to create because he runs a background thread to new the regex, which should have told him something. We'll fix the code that uses the regex to do something saner, soon.) All that aside, I've been happy with the 64-bit version of Windows 7. [...]



When Video Cards Go Bad

Sat, 20 Jun 2009 04:50:00 GMT

(image)

I complained a week ago about my display driver going berserk. I blamed Windows Update, since it happened within hours of a pile of updates being installed. I upgraded to the latest beta NVidia drivers on Monday and it helped for a while, but by Wednesday, it was almost as bad again as it had been last Friday. It was infuriating and I was both entertaining and alarming my neighbors with my cursing.

Today was the last day of a very busy sprint for me and at last I had the time to dig into it. I opened up the case and took a look at both video cards—I have two dual-head cards connected to three monitors—and one of them had partially blown capacitors like those in the picture. I removed the bad card and did some graphics-intensive things for an hour, and the other card behaved flawlessly.

Oddly, until someone mentioned that it might be a hardware problem yesterday, it didn't occur to me, even though a video card blew in this machine last year. I came in one morning to find a black monitor, and when I pulled out that card, I found that some of the capacitors had popped right open with stuffing protruding.

On general principles, I had been meaning to repave this machine for a while. I've had it since December 2007 and it was still running the original installation of Vista. I booted from a DVD, reformatted my C: drive, and installed Windows 7 x64 RC1.

I finally have a 64-bit OS as my primary Windows desktop, so I'll actually be using the Win64 build of Vim that I maintain. My first impressions of Windows 7 on this machine are very favorable, but there's plenty more that I need to install before the machine has everything that I need.




Display driver nvlddmkm stopped responding and has successfully recovered

Sat, 13 Jun 2009 07:34:00 GMT

(image)

This morning, the video adapters on my Vista dev box were resetting 2–3 times per minute.

After a pile of Windows Updates landed on my machine at 3am yesterday, it would occasionally freeze solid for a few seconds. Once in a while, all the monitors would go black briefly, then restore. Each time, I would see a status update pop up from the system tray, "Display driver nvlddmkm stopped responding and has successfully recovered."

This was irritating enough that I downloaded the latest NVidia drivers this morning, 185.85_desktop_winvista_32bit_english_whql.exe. That really screwed me. The video adapters started resetting 2–3 times per minute, rendering the machine almost unusable. I have two video adapters, NVidia GeForce 8600 GT and NVidia GeForce 7600 GT.

The eventlog was full of Event ID 4101 - Display Driver Timeout Detection and Recovery.

I reverted to the 178.24 drivers and that helped. When I'm not touching the machine, the adapters only get reset every few minutes instead of several times a minute. When I am using it, something as simple as clicking a window to bring it to the foreground can trigger a reset.

It's very irritating but I can live with it for a little while, unlike the other. I don't want to repave my box: apart from the time loss, I'm not convinced that it would help if I got the same driver config all over again.

I contacted a friend at Microsoft who tried to hook me up with a driver guy, who is unfortunately out of office. I'm hoping that it can be fixed early next week or my temper is going to fray rapidly.

Update: June 19th: See When Video Cards Go Bad.