Subscribe: /* Rambling comments... */
http://www.lenholgate.com/rss2.xml
Added By: Feedage Forager Feedage Grade A rated
Language: English
Tags:
bug  code  fix  framework  jetbytetools  new  release  server framework  server  studio  time  version  visual studio  visual 
Rate this Feed
Rate this feedRate this feedRate this feedRate this feedRate this feed
Rate this feed 1 starRate this feed 2 starRate this feed 3 starRate this feed 4 starRate this feed 5 star

Comments (0)

Feed Details and Statistics Feed Statistics
Preview: /* Rambling comments... */

www.lenholgate.com - Rambling Comments - Len Holgate's C++ progamming blog



/* Rambling comments... */ Len Holgate's thoughts on this and that... Mainly test driven software development in C++ on Windows platforms...



Updated: 2016-11-15T16:29:47Z

 



C++ Tools - JetBrains ReSharper C++ - purchased... - Rambling Comments

2016-11-15T16:29:47Z

I've been looking at Resharper C++ by JetBrains for a while now and the trial period has finally run out. I immediately bought a license which shows how my feelings have changed about the product during the trial....

I've been looking at Resharper C++ by JetBrains for a while now and the trial period has finally run out. I immediately bought a license which shows how my feelings have changed about the product during the trial.

Initially I found that the tool got in my way too much, I still find it a little sluggish at times but I think that my initial tests were hurt by the fact that I was running multiple copies of Visual Studio (as is my way) with multiple large projects and generating code inspection reports in all of them at the same time... Well, that's how I'd want to use it... Anyway, when limiting myself to one or two concurrent instances things were better. There's still a bit of a lag before the 'light bulb' icon appears after you navigate from the inspection report to the editor and I suspect that this is because the addin has to run the inspection again for the newly opened file before knowing what's wrong with it...

I haven't really explored much functionality other than the code inspection reports but these are a good complement to my other static analysis tools and the refactoring capability of the addin is interesting. I'm looking forward to trying the command line version of the inspection report and hopefully that can be integrated into my CI server's build process.

As I've said from the start, the support is excellent and I liked the fact that the trial period could be paused; though I found out about it too late to actually pause it. Too often I install trial editions of tools only to find that I'm dragged away to do some client work and by the time I get back to the tool the trial has expired. The price is reasonable, the licensing is flexible and the online purchase experience was painless.

(image)



C++ Tools - CppDepend - Rambling Comments

2016-11-14T10:08:02Z

I've been trying various static analysis tools on the C++ code of The Server Framework. So far I'm using Resharper C++ and the Gimpel PC-Lint Plus Beta on a regular basis and I've now added CppDepend into the loop. Full...

I've been trying various static analysis tools on the C++ code of The Server Framework. So far I'm using Resharper C++ and the Gimpel PC-Lint Plus Beta on a regular basis and I've now added CppDepend into the loop.

Full disclosure. I have been given a CppDepend license.

As I've said before, whilst CppDepend is easy to get hold of, easy to install and "just works" I don't find it that useful. I can certainly remember large enterprise clients where this kind of tool would be invaluable for management level analysis of large codebases but for a small development team of competent people it's less immediately useful. That said, I've found several warnings that it produces to be helpful and so I've been running it alongside the other tools as it fills some gaps.

Since I'm in the process of dropping support for several old compilers I can finally begin to move the codebase forward to slightly more modern C++. All of the tools help with this and CppDepend has some 'modernise C++' checks that I'm finding useful.

I like the fact that I can run CppDepend as a stand alone GUI. I prefer this method to using fully integrated Visual Studio extensions.

I like the idea of the regression reports but haven't actually set up a baseline report and run them...

I'm not sure that I would purchase a license for this tool but I know clients that could benefit from using it.

(image)



Practical Testing: 37 - Bringing the code up to date - Rambling Comments

2016-11-09T14:01:31Z

Previously on "Practical Testing"... Having just resurrected this series of blog posts about testing a non-trivial piece of real-world C++ code I've fixed a few bugs and done a bit of refactoring. There's one more step required to bring the... Previously on "Practical Testing"... Having just resurrected this series of blog posts about testing a non-trivial piece of real-world C++ code I've fixed a few bugs and done a bit of refactoring. There's one more step required to bring the code in this article right up to date. The timer queue that is the focus of these blog posts is part of The Server Framework. This body of code has been around since 2001 and has evolved to support new platforms and compilers. One of the things that I do from time to time is remove support for old platforms and compilers. This allows me to start using exciting new C++ features (around 10 years after they were first 'new') and it means that code that is present just to work around issues in old platforms can be removed. The timer queue has quite a bit of code that needn't be there now that Windows XP has passed away. In 2008, in episode 17, I added support for GetTickCount64() which removed some of the complexity from timer management. We kept the old version around as you needed to have Windows Vista or later to use that API call. Now that I no longer support Windows XP every supported platform has GetTickCount64() and there's no reason to support the XP version. Removing that removes quite a bit of complexity from both the internals and the public API; you no longer have to decide which version to use as there is only one! As the recent bug fixes have shown, there are also two versions of timer dispatch. The original version which holds a lock whilst calling into user code during timer dispatch and the improved version that doesn't. Holding locks whilst calling back into user code via callback interfaces is a surefire way to create lock inversions and deadlock. The old dispatch method was originally kept because I was unsure of the performance characteristics of the new way. The improved version was added in 2008 in episode 18 and every piece of development that I've done since has used the new dispatch method. The performance difference, if present, is not relevant and the removal of the ability for code to take part in lock inversions is important for library code. So the original method of dispatch should be removed. The removal of this functionality massively simplifies the code and the tests. 60 tests can be removed which is almost a third. Most of the code that was moved into the common base class for the two versions of the queues can be moved into the 'Ex' version with GetTickCount64() support. I expect that I should, eventually, rename the CCallbackTimerQueueEx class to CallbackTimerQueue, but not yet. There are some systems that use this class explicitly rather than via CThreadedCallbackTimerQueue and the enum that we're removing in this release. It will be easier to introduce another breaking change in a future release as it means that fixing the breaking changes due to this release is slightly easier and more obvious and the fix when I do change the name will be a simple rename in the code that uses the old class... The code is here on GitHub and new rules apply. [...]



C++ Tools - JetBrains ReSharper C++ is slowly winning me over - Rambling Comments

2016-11-12T19:51:22Z

I've been looking at Resharper C++ by JetBrains for a while now and I expect I'm nearing the end of the trial period. Initially I found it got in my way but slowly I think it's training me to ignore... I've been looking at Resharper C++ by JetBrains for a while now and I expect I'm nearing the end of the trial period. Initially I found it got in my way but slowly I think it's training me to ignore the niggles and I'm finding the functionality quite compelling. At present, I'm most interested in the "code inspection" report from Resharper. Comparing it to the similar functionality in PC-Lint and CppDepend, etc. As such the integration with the Visual Studio editor was, initially, of less interest to me. With the code inspection functionality you can run a report and then work your way through a list of issues rather than just seeing icons pop up as you're navigating in the editor. With the ability to suppress warnings that you don't care about this kind of "list of issues" approach is useful for spotting regressions and examining large code bases.[...]



Practical Testing: 36 - Timeout handle wrap - Rambling Comments

2016-11-09T14:02:05Z

Previously on "Practical Testing"... I've just fixed a new bug in the timer queue and in doing so I updated the code used in the blog posts to the latest version that ships with The Server Framework. This code included... Previously on "Practical Testing"... I've just fixed a new bug in the timer queue and in doing so I updated the code used in the blog posts to the latest version that ships with The Server Framework. This code included two fixes that had been fixed some time ago and which I hadn't documented here. They also lacked unit tests... Last time, I wrote tests for, and fixed, the first bug. This time I fix the final bug. This bug is in the "lock-free" timeout handling code and it would cause the threaded version of the timer queue to do a small amount of unnecessary work but otherwise work correctly. The problem is that we use a ULONG_PTR value as an opaque handle when processing timeouts using the "lock-free" BeginTimeoutHandling(), HandleTimeout(), EndTimeoutHandling() sequence and this opaque handle has a sentinel value of 0 that is used to indicate that there are no timeouts to handle. The code that we use to generate the timeout handle in BeginTimeoutHandling() is a simple ::InterlockedIncrement() and, if enough timeout handles are generated, this will wrap to 0 and return the sentinel value when it should be returning a non-sentinel value. Making the problem visible Initially when I fixed this issue I did so without adding a test as I felt that it was too complex to add some method for the test code to manipulate the timeout handle generation to ensure the bad value was generated and running a test which called BeginTimeoutHandling() enough times to cause the bug to manifest would be too slow. The fix is easy, seeing the problem is less so. Whilst looking at the code again and thinking about how I could justify not needing a test here I realised that it was trivial to make this issue very visible. Rather than starting the timeout handle value as the address of the object (don't ask, we'll refactor it later on) instead we set the initial value to -1. The first time we handle timeouts we will increment the value to zero and the bug will be exposed. Fixing it Fixing this is trivial, rather than simply calling InterlockedIncrement() we call it and check that we don't have our sentinel value. If we do, we call it again... This code does the job: IManageTimerQueue::TimeoutHandle CCallbackTimerQueueBase::GetNextTimeoutHandle() { TimeoutHandle timeoutHandle = ::InterlockedIncrement(reinterpret_cast(&m_nextTimeoutHandle)); if (timeoutHandle == InvalidTimeoutHandleValue) { // wraps to zero? this is possible, avoid it... timeoutHandle = ::InterlockedIncrement(reinterpret_cast(&m_nextTimeoutHandle)); } return timeoutHandle; } The code is here on GitHub and new rules apply. Does it really need to be this complex? Whilst the changes expose the issue in a way that can be tested and then fix the problem I'm not especially happy with them. The code has obviously evolved from something that returned an opaque representation of a pointer to something that just returns a number... Looking back at the blog post where this concept was introduced, and looking at the code, I can see that originally we returned the timeouts themselves via the BeginTimeoutHandling() call and so it was essential to pass in a valid handle (pointer to timeouts) to the other functions as they dealt with the timeouts and tidied them up. When the code was changed to use intrusive containers in episode 32 the timeout handle changed and was no longer an opaque pointer to timeouts but instead became a meaningless value that just had to match in the calls to HandleTimeout() and EndTimeoutHandling(). It's likely a better design to simply return a boolean from BeginTimeoutHandling() and remove the entire concept of a timeout handle. The timer wheel implementation still uses the 'return an op[...]



Practical Testing: 35 - Heap corruption - Rambling Comments

2016-11-08T15:32:16Z

Previously on "Practical Testing"... I've just fixed a new bug in the timer queue and in doing so I updated the code used in the blog posts to the latest version that ships with The Server Framework. This code included... Previously on "Practical Testing"... I've just fixed a new bug in the timer queue and in doing so I updated the code used in the blog posts to the latest version that ships with The Server Framework. This code included two fixes that had been fixed some time ago and which I hadn't documented here. They also lacked unit tests... In this episode I find and fix the first of these issues by writing a unit test that triggers the issue. This bug is another edge case that isn't used that often by most of the code that uses the timer queue. The queue supports timers that can be set and reset a number of times via an interface that allows you to create a timer, set/cancel it and then destroy the timer. A less used interface allows you to create 'fire and forget' timers that can only be set once and that clean themselves up. Hardly any code that I write uses this interface but it's there for backwards compatibility and the code required to support it is limited to the call that sets the initial timer, as the cleanup is done by code that's shared with timers that are deleted during their own timeout processing. This bug also only affects the Timer Wheel implementation which has a considerably smaller set of users and a considerably narrower use case. There's test coverage for "one shot" timers but only for timers that are processed whilst the timer queue's internal lock is held. There is no test for a "one shot" timer for queues that dispatch without holding a lock. The code for lock-free dispatch is significantly different to the code for dispatch whilst holding a lock and that's where the bug is. The bug was originally found because it causes heap corruption due to a double delete. The first thing I'll do is add a test for lock-free timer dispatch of "one shot" timers. This clearly demonstrates the bug when run in release mode on Windows 10 as the heap alerts us to the problem. Adding a new test The new test is similar to TestOneShotTimer() in TCallbackTimerQueueTestBase, which contains tests that run for all timer queue implementations. The TestOneShotTimer() test does everything needed to show that a "one shot" timer works but it only tests the code that dispatches timers using HandleTimeouts() to dispatch the timers. We need a test that uses the "lock-free" BeginTimeoutHandling(), HandleTimeout(), EndTimeoutHandling() sequence. That code looks something like this: template void TCallbackTimerQueueTestBase::TestOneShotTimer() { JetByteTools::Win32::Mock::CMockTimerQueueMonitor monitor; P tickProvider; tickProvider.logTickCount = false; { Q timerQueue(monitor, tickProvider); CheckConstructionResults(tickProvider); THROW_ON_FAILURE_EX(INFINITE == timerQueue.GetNextTimeout()); tickProvider.CheckNoResults(); CLoggingCallbackTimer timer; const Milliseconds timeout = 100; timerQueue.SetTimer(timer, timeout, 1); CheckTickProviderSetTimerResults(tickProvider); const Milliseconds expectedTimeout = CalculateExpectedTimeout(timeout); THROW_IF_NOT_EQUAL_EX(expectedTimeout, timerQueue.GetNextTimeout()); tickProvider.CheckResult(_T("|GetTickCount|")); timerQueue.HandleTimeouts(); tickProvider.CheckResult(_T("|GetTickCount|")); timer.CheckNoResults(); THROW_IF_NOT_EQUAL_EX(expectedTimeout, timerQueue.GetNextTimeout()); tickProvider.CheckResult(_T("|GetTickCount|")); tickProvider.SetTickCount(expectedTimeout); THROW_ON_FAILURE_EX(0 == timerQueue.GetNextTimeout()); tickProvider.CheckResult(_T("|GetTickCount|")); timerQueue.HandleTimeouts(); tickProvider.CheckResult(_[...]



C++ Tools - Some thoughts on JetBrains ReSharper C++ - Rambling Comments

2016-11-09T09:06:44Z

Following on from my rant about C++ tooling and its follow up. I've been looking at JetBrains ReSharper for C++. This isn't a review, more just my initial thoughts. TL;DR I'd like to like it. It does some good things...

Following on from my rant about C++ tooling and its follow up. I've been looking at JetBrains ReSharper for C++. This isn't a review, more just my initial thoughts.

TL;DR I'd like to like it. It does some good things but it also gets in my way and slows me down.

ReSharper is a Visual Studio addin. In general I don't like addins but this comes from my years working short contracts and multiple clients where it was easiest to be at home in a clean Visual Studio installation as no two clients would have the same addins installed. ReSharper's adding intercepts lots of standard Visual Studio functionality and I'm sure that I could get used to it but intellisense and stuff is fractionally slower than without it installed (and yes, I do have a machine that well exceeds the basic hardware requirements). It also does some things just differently enough that it means I automatically do the wrong thing and then have to redo the right thing. This interrupts my flow. I'm sure that if I spent enough time with it I'd get used to it but I'm not sure I will spend enough time with it.

Right now I'm most interested in the inspection side of ReSharper. Inspecting a solution takes quite a while, which is to be expected. Some of the coding violations that it reports are useful, some less so, some inappropriate, but that's always the way with these tools. It seems to be difficult to suppress violations. I like the way PC-Lint allows for inline suppression comments (which I hate, but use anyway) and external suppression files. I work on lots of different codebases and common suppression rules are a must and these MUST be project/work item specific.

I've had really good support from JetBrains. The addin popped up a message box telling me I was half way through my trial and asking me how I was getting on. I responded with some of my gripes and had an email within a day from a helpful support person with some work arounds and explanations.

I'm still using it, but I'm not sure I'll continue with it after the trial completes.

(image)



Practical Testing: 34 - Potential reentrant locking deadlock - Rambling Comments

2016-11-08T14:31:57Z

Back in 2004, I wrote a series of articles called "Practical Testing" where I took a piece of complicated multi-threaded code and wrote tests for it. I then rebuild the code from scratch in a test driven development style to... Back in 2004, I wrote a series of articles called "Practical Testing" where I took a piece of complicated multi-threaded code and wrote tests for it. I then rebuild the code from scratch in a test driven development style to show how writing your tests before your code changes how you design your code. Since the original articles there have been several bug fixes and redesigns all of which have been supported by the original unit tests and many of which have led to the development of more tests. In the previous update in July 2014 I added custom intrusive containers to replace the stl containers and reduce memory allocation contention and improve performance. Since then the code has been in regular use by clients and users of The Server Framework and has stayed pretty stable. Recently a client has upgraded some rather old code to a new version of The Server Framework and triggered a bug that has lain dormant in the timer queue since release 6.6 of The Server Framework when I added some new locking classes. Most clients haven't triggered this bug because they don't use the code in this particular way. So, in this installment I detail the bug and present some tests which reveal it. The fix is pretty simple, we simply have to change the lock type used by the threaded timer queue so that it allows reentrant locking. A non-reentrant lock The CLockableObject class is a replacement for my CCriticalSection both provide locking functionality but the CLockableObject uses a SRWL (Slim Reader Writer Lock) for exclusive locking which works out to be faster than using a CRITICAL_SECTION. The downside of the SRWL is that it's not a reentrant lock and if you try and obtain it whilst you're already holding it you will deadlock. The functionality that we require for the fully featured CLockableObject class includes the ability to "TryEnter" and as this wasn't available until after Windows Vista we have some conditional compilation that degrades the implementation of a CLockableObject back to using a CRITICAL_SECTION if the required functionality isn't available. This means that you can switch to using the new lock type sooner and reap the rewards when you build for the correct version of Windows... The downside of this flexibility is that you may find that your new lock is used in a reentrant fashion when it shouldn't and it doesn't cause problems because it's using the reentrant CRITICAL_SECTION implementation rather than the full featured SRWL version... TL;DR My new lock type is supposed to be non reentrant but sometimes it isn't. Code that works when it isn't will fail when it is. Since many users of the framework don't have as much unit testing as they should I added some functionality to the CLockableObject so that it could warn of attempts at reentrant use. This means you can start using it and replacing some of your CCriticalSections with it with the reentrancy checks enabled and you'll get warnings at run-time if the lock you've replaced WAS used in a reentrant fashion after all... The bug Early on in this series of articles I pointed out that the threading of the timer queue was orthogonal to the timer management. To that end the threading was split into a derived class, CThreadedCallbackTimerQueue this is the class that is thread safe and contains the lock in question. Originally this class used to hold its lock during timer dispatch but this was a source of lock inversions and so changes were applied to allow you to dispatch without holding the lock. This complicated the dispatch somewhat and reduced performance a little and so the two options were available so that you could decide what w[...]



More thoughts on C++ Tools (on Windows) - Rambling Comments

2016-10-15T12:12:36Z

Last friday's rant about C++ static and dynamic analysis tools was picked up by Reddit and I have had quite a few helpful suggestions for other tools to try. Thanks! Dr. Memory. Microsoft's Application Verifier. CPPDepend PVS Studio Microsoft's CRT...

Last friday's rant about C++ static and dynamic analysis tools was picked up by Reddit and I have had quite a few helpful suggestions for other tools to try. Thanks!

  • Dr. Memory.
  • Microsoft's Application Verifier.
  • CPPDepend
  • PVS Studio
  • Microsoft's CRT Debug heap.
  • CLang - various options, especially using AddressSanitizer.

So far I've had a quick look at Dr. Memory, which seems to be pretty good. Free, easy to download and install and it's faster than DevPartner and 'just works'. Suppressing errors that aren't really errors is easy. Running it is easy and it feels like it will just fit into my work process without me having to conform to its idea of how I should be working. I can see me integrating it into my build server's workflow and running all of my tests through it. So far it has found a few obscure leaks for me; locale leaks (!) in some string format conversion code that nothing else has noticed. To me it feels like the tool BoundsChecker should have been and it doesn't have a pointless GUI, it's just a command line tool that does the job and produces easy to understand reports.

CppDepend also 'just works'. It's easy to get hold of a trial version and it installs and runs without issue. I've used it before and whilst it does spot some of the issues that something like Gimpel PC-Lint will spot it's much more geared towards enforcing corporate coding guidelines. Yes it's very configurable, but, out of the box it doesn't give me enough static analysis and gives me too much of other people's style rule infringements.

Microsoft's Application Verifier is better than I remember it being the last time I tried it (many years ago) but it's not as usable as Dr Memory.

I haven't downloaded PVS Studio as I figure that it doesn't really matter how good it is, the licensing is of the "we don't tell you how much it is until you ask us" variety of Enterpricey and the minimum license is for a team of 9...

The CLang options looks interesting but I haven't had the time to investigate fully yet.

So thank you Reddit. I now have a tool in my tool box that should help with my current client problems and several other tools to explore.

(image)



It must be about time for a "the state of C++ tooling" rant... - Rambling Comments

2016-10-12T11:44:25Z

I tend to develop code with JITT (Just in time testing), this is like TDD when I'm doing it but it doesn't always get done. What does get done, generally, is the "first test" for each class. This ensures that... I tend to develop code with JITT (Just in time testing), this is like TDD when I'm doing it but it doesn't always get done. What does get done, generally, is the "first test" for each class. This ensures that subsequent testing is possible as the code doesn't have hidden dependencies and it gives me a test harness that's ready to go when I find that I need it. More complex code end up being TDD, easier bits end up as JITT where the tests are only written when I've wasted time banging my head on the desk trying to debug problems the "old fashioned way". This approach tends to limit the number of really horrible bugs (except race conditions!) that I have to deal with and I can't remember the last time that I had memory corruption issues etc. Client code, that is code that my client's write, tends to be different and every so often I'll have a client who has got themselves so twisted up with 'weirdness' that they ask me to track down their memory issues for them. At this point I root around in my tool box for the 'memory validation' and 'code analysis' tools and dust them off. I then spend a few moments in that heavenly state that you can get into when you believe the hype about something, before coming down to earth with a bang and writing one of these "state of C++ tooling" rants. Let's start with the good experience.... The first tool I decided to try was static analysis with Gimpel Lint using Visual Lint to run it as it's the only way to be productive with the tool. Unfortunately Gimpel Lint doesn't really like modern C++. There's a beta program for a new version which DOES play nicely with modern C++ but I hadn't really used it that much as I didn't need it for The Server Framework code as we have to deal with some very old compilers and so the code tends to use older C++ functionality. My client's projects were VS2015 and running the Visual Lint on them generated PC-Lint config files that warned of incompatibilities between the version of PC-Lint I was using and the VS2015 header files... The linting failed spectacularly due to the content of the compiler's header files, specifically the newer C++ stuff. I was a little confused because I'd run the tools on my framework code in VS2015 with no problems. I then noticed that when run on the framework code, Visual Lint generated a config file for VS2012 rather than 2015 and used the 2012 headers. This allowed PC-Lint to work but wasn't right. A few tweets to Anna-Jayne and Beth at Riverblade and they'd worked out what was going on (I have some wacky Visual studio projects that have been upgraded through every possible compiler release and these contain some artifacts from VS2012 which Visual Lint is recognising and which cause it to generate the wrong config). A fix is on the way, which is brilliant and as responsive as I've come to expect from them. Unfortunately, all that the fix to Visual Lint will do is mean that PC-Lint will fail on ALL of my VS2015 projects rather than just some. The real fix for this issue is to use the PC-Lint Plus beta which should understand the VS2015 C++ headers. Well the beta is a beta. It's not really ready for prime time, at least not on my client's codebase and I can't run it on the 'clean' code of my framework release as I'm waiting on the Visual Lint fix. The mediocre... I then configured Visual Lint to run CppCheck which is an OpenSource C++ static analysis program. That ran and gave the framework code a pretty clean bill of health, but in a way, it was too clean. With PC-Lint I've had to annotate the code with annoying comment warts to tell it to let me work outside of i[...]



Hosting .Net Core in C++ - Rambling Comments

2016-09-30T10:19:08Z

One of the things that came out of my conversations with clients last night was an interest in hosting .Net Core from native code. Of course we already host the CLR and provide an easy way to write servers...

One of the things that came out of my conversations with clients last night was an interest in hosting .Net Core from native code.

Of course we already host the CLR and provide an easy way to write servers that do the heavy lifting in native code and call out to managed code for the business logic. We have several clients using this to host managed "plugins" inside a native host and it works very well.

Hopefully hosting .Net Core won't be that different. It looks like you can skip the whole meta host thing and just load it up directly from a DLL. More details as we explore...

(image)



London MMO Meetup - Christof Wegmann from Exit Games and Ben Hollis from King - Rambling Comments

2016-10-01T09:22:22Z

I dragged myself into London last night for the London MMO Meetup. I had some clients that I wanted to chat to who were going and the programme looked good. This was the first "meetup" that I'd been to,...

I dragged myself into London last night for the London MMO Meetup. I had some clients that I wanted to chat to who were going and the programme looked good.

This was the first "meetup" that I'd been to, it was good and the format worked well. We were hosted by King at their London office on Wardour Street. The office was great, the presentation space was good and they provided a nice spread of food and drink.

Ben Hollis from King spoke first about the considerations of designing a casual, synchronous, multi-player game and the design decisions involved in insuring that people actually want to play and keep playing. It was fascinating stuff for a server dev like myself who rarely needs to think about this stuff even when my clients are gaming people. The biggest take away for me was how King tracks player activity, such as how many players progress to each level, etc. and uses this for feedback into how the game is designed. I'm always telling my clients that getting good data out of your server is important but my focus is usually on performance and server design issues rather than player/client interaction stuff. I can see that I'll be pushing even harder for people to understand the importance of getting lots of data from your server so that you can visualise what's going on.

Next up was Christof Wegmann from Exit Games who had a more technical talk about how 'netcode is hard', tell me about it... His guys had an interesting take on multi-player games and how more modern (and complex) forms of mult-user network interaction are unnecessary for some types of game and how something a little simpler (and more old school) can work. He then went on to detail his company's take on Deterministic Lockstep and how they'd overcome and replaced some of the non-deterministic APIs in Unity to allow them to implement a deterministic lockstep protocol which avoids lag with a 'guess ahead' algorithm and then uses an intelligent rollback technique if the guessing was incorrect. It all sounded pretty clever and for the game genres where it's appropriate it looks like it should be much smoother than a more normal deterministic lockstep design.

Thanks to Joe Nash for organising and Viviane Costa from King for hosting.

(image)



Supporting Visual Studio 2015 Update 3 - AsynchronousEvents

2016-07-11T16:04:10Z

Visual Studio Update 3 adds some new warnings for precompiled header generation. One of these, C4598, will prevent precompiled header builds of framework code due to our warnings as errors policy. A fix for this issue is to add the...

Visual Studio Update 3 adds some new warnings for precompiled header generation. One of these, C4598, will prevent precompiled header builds of framework code due to our warnings as errors policy.

A fix for this issue is to add the new warning to the list of disabled warnings in the project file. The easiest way to do this is to do a search and replace across your source tree for *.vcxproj files and replace "4627;4654;" with "4627;4654;4598;".

I need to work out what the implications of this compiler change are on how our precompiled headers are generated.

(image)



Latest release of The Server Framework: 6.7 - AsynchronousEvents

2016-12-01T19:35:01Z

Version 6.7 of The Server Framework was released today. This release is mainly a code simplification release which removes support for legacy compilers and operating systems. See here for more details. However, there are some breaking changes where smart... Version 6.7 of The Server Framework was released today. This release is mainly a code simplification release which removes support for legacy compilers and operating systems. See here for more details. However, there are some breaking changes where smart buffers have replaced buffer references and this causes function signature changes. In addition there has been considerable churn in the Streaming Media Option Pack with knock on changes in the HTTP library code which needed to be made more complete to deal with serving HLS streams. As always, see the release notes here, for full details of all changes. Breaking changes: Breaking change JetByteTools::IO::CAsyncFileReader, JetByteTools::IO:: CAsyncFileWriter, JetByteTools::IO::CAsyncFileWriterEx and JetByteTools::IO::IAsyncIOStream now work in terms of JetByteTools::IO:CSmartBuffer where possible. This potentially reduces the reference counting activity on the buffers. Breaking change Where possible JetByteTools::IO::CSmartBuffer has replaced raw pointers and references to buffers. This massively reduces the need to reference count buffers during normal I/O operations and increases performance, especially on NUMA architectures. Bug fixes: Bug fixes to JetByteTools::IO::CBuffer around the usage of m_maxBytesToRead. Bug fix to JetByteTools::Win32::CCallbackTimerQueueEx::BeginTimeoutHandling() to prevent incrementing m_nextTimeoutHanlde causing the value to wrap to InvalidTimeoutHandleValue which was possible, but unlikely. Changes: Dropped support for Visual Studio 2005 and Visual Studio 2008. Dropped support for Windows XP. Removed JETBYTE_HAS_INTERLOCKED_64 as it's true on all supported platforms now. Removed JETBYTE_WARN_ON_SOCKADDR_STORAGE_DEF and JETBYTE_WARN_ON_WSPIAPI_COUNTOF_DEF as these are no longer relevant. Removed JETBYTE_PERF_STREAM_SOCKETS_SKIP_EVENT_ON_HANDLE, JETBYTE_PERF_DATAGRAM_SOCKETS_SKIP_EVENT_ON_HANDLE and JETBYTE_PERF_FILE_WRITER_SKIP_EVENT_ON_HANDLE as these are always enabled now. Removed JETBYTE_PERF_STREAM_SOCKETS_SKIP_MARSHAL_TO_IO_POOL and JETBYTE_PERF_DATAGRAM_SOCKETS_SKIP_MARSHAL_TO_IO_POOL as these are always enabled now. Marshalling was only required to work around Windows XP's I/O cancellation on thread termination policy. Removed JETBYTE_DEPRECATE_SHARED_CRITICAL_SECTIONS, JETBYTE_DEPRECATE_SHARED_LOCK_SOCKETS, JETBYTE_DEPRECATE_CRITICAL_SECTION_2, JETBYTE_DEPRECATE_LOW_CONTENTION_BUFFER_ALLOCATOR, and JETBYTE_DEPRECATE_TLS_BUFFER_ALLOCATOR as these features are no longer available. Removed SecureCRT.h as it's no longer required. It was used to map differences between the Visual Studio 2005 CRT and the "new" secure CRT. Added the macro, SuppressLNK4221Warning(), which can be put into a file in order suppress the MS Visual C++ Linker warning 4221 - "warning LNK4221: no public symbols found; archive member will be inaccessible" Turn off JETBYTE_INTRUSIVE_RED_BLACK_TREE_INTERNAL_STATE_FAILURE_EXCEPTIONS by default. Removed JetByteTools::IO::CLowContentionBufferAllocator and JetByteTools::IO::CTLSBufferAllocator. Added JetByteTools::IO::CBufferBasedBufferAlloctor which is a simple shim to allow an instance of JetByteTools::IO::IBuffer to be used as an implementation of JetByteTools::IO::IAllocateBuffers. Removed JetByteTools::IO::IIOPool::DispatchToAll() as it was only required for issuing CancelIO() calls to all I/O threads on XP where CancelIOEx() wasn't available. Added an overload of JetByteTools:[...]



Visual Studio Component Cache bug looks like it's here to stay... - Rambling Comments

2016-06-08T11:11:07Z

So, it's nearly a year since I first started noticing issues with VS2015 on my build servers. The bug in question now has an entry on Microsoft Connect and Google can help with some work arounds which don't require turning...

So, it's nearly a year since I first started noticing issues with VS2015 on my build servers. The bug in question now has an entry on Microsoft Connect and Google can help with some work arounds which don't require turning the machine off and on again a random number of times until it works... There's even a Visual Studio extension that fixes the issue for you (!).

I find it disappointing that this hasn't been fixed, it's a fundamental usability issue which seems to be causing lots of people lots of pain. It's probably not too bad if you're running Visual Studio as a developer in the 'normal' way; especially if the extension can fix the issue for you when it happens, but on a build machine it's a pain. Of course, it only ever happens just after you kick off a build and leave the office. If you're sitting there waiting for it to happen the problem never seems to manifest...

(image)



6.7 - Potentially faster code, in some circumstances... - AsynchronousEvents

2016-06-25T10:58:17Z

I hinted at the end of the last post that the 6.7 release might increase performance a little. Well, whilst the bulk of the changes in 6.7 are purely code cleaning and the removal of legacy support there is a...

I hinted at the end of the last post that the 6.7 release might increase performance a little. Well, whilst the bulk of the changes in 6.7 are purely code cleaning and the removal of legacy support there is a fairly major functional change as well.

In most situations references or pointers to I/O buffers have been replaced with smart pointers. This change may cause some issues during an upgrade as you need to change some function signatures from IBuffer refs to CSmartBuffers. The advantage is that in many servers there will no longer be any need for buffer reference counting during normal I/O operations.

The Server Framework relies on reference counting to keep the objects that are used during the asynchronous operations alive until those operations complete. So we increment a counter on the socket object and also on the buffer object when we initiate an operation and then decrement the counters when the operation completes. I'm sure there are other ways to manage the object lifetime but this has worked well for us.

The problem is that these increments, although they look like cheap operations, can be quite expensive, especially on NUMA hardware.

Whilst there's not much we can do about the reference count on the socket object, the buffer doesn't really need to be reference counted most of the time. Or, more's the point. The initial reference can (and should) be passed along rather than each stage taking and releasing its own reference. With a buffer you generally only want to be accessing it from one thread at a time and so you allocate it and then issue an operation and pass the reference you have off to the operation. When the operation completes the code captures the reference and takes ownership of it and then the buffer can be processed. If you're lucky you can then use the same buffer for a response to the operation and pass it back to the framework again.

This requires a few changes to your code but it's fairly straight forward. Your OnReadCompleted() handler will give you a CSmartBuffer and if you want to retain ownership of it after the handler returns then you simply need to detach the buffer from the CSmartBuffer you were given.

This is only "potentially faster" as it really depends on the structure of your server and how you deal with our I/O buffers but the framework is no longer standing in the way of this kind of optimisation, and we've removed a couple of reference adjustments in the normal operation flow.

(image)



Another release is coming... - AsynchronousEvents

2016-06-25T16:06:14Z

We've only just shipped Release 6.6.5 of The Server Framework but we already have another release that's just about to ship. This isn't because some horrible bug has slipped through our testing, it's because we've been planning to produce a... We've only just shipped Release 6.6.5 of The Server Framework but we already have another release that's just about to ship. This isn't because some horrible bug has slipped through our testing, it's because we've been planning to produce a 'clean up' release for some time. 6.7 is that release. Lets be straight here, 6.7 is a release for us more than for you. The aim is to simplify our build/test and release process, remove dead code whilst introducing no new bugs and removing no functionality that you rely on. So what does 6.7 achieve. Well, for a start we drop support for Visual Studio 2005 and 2008 and also for Windows XP. Removing support for these legacy compilers and operating systems means that we can remove all the code that was required just to support them. This massively simplifies our code base without removing anything that the code actually relies on to run on modern operating systems. Windows Vista introduced massively important changes to asynchronous I/O and we have supported these changes for a long time (over 8 years!). The code required to jump through hoops to make code running on Windows XP behave was complex. For example, Windows XP would cancel outstanding I/O requests if the thread that issued them exited before the I/O request completed. We had a marshalling system in place to ensure that I/O operations were only ever executed on threads that we controlled so that you'd never be faced with unexpectedly cancelled operations. All of that can go now. Removing XP also means we no longer need to maintain an XP machine in our build farm. It's one less configuration that needs to be built and tested before a release. Dropping support for VS2005 and 2008 removes 4 complete sets of builds (x86 and x64 for each compiler) plus all of the conditional code that was required to support the older compilers. At last we can start moving towards a slightly more modern C++, perhaps. Some old code has been removed; there's no need, on modern operating systems, to share locks. This worked really well back in the day, but, well, we were running on Windows NT at the time and resources were much more limited than they are now. All of the "Shared Critical Section" code is now gone. This has knock on effects into the Socket Tools library where all of the shared lock socket code has been removed. Nobody should be using that in 2016 anyway! You can no longer set a critical section's spin count in the socket allocator, it never really worked anyway as the lock was used for too many different things. Some experimental code has also been removed; The TLS and Low Contention buffer allocators are gone. The horrible "dispatch to all threads" cludge has been removed from the I/O pools (it was only there to support pre-Vista CancelIO() calls which are no longer needed now that we have CancelIOEx()). The original callback timer queue that was based on GetTickCount() and which spawned Len's "Practical testing" series of blog posts (back in 2004!) has gone. There's no need for the complexity when all supported operating systems have GetTickCount64(). Finally we've slimmed down our set of example servers. Removing servers which didn't add much value or which duplicated other examples. Again, this speeds our release process by speeding up the build and test stage as there are fewer servers to build and fewer tests to run. So, what's in it for you? Well, a fast[...]



Latest release of The Server Framework: 6.6.5 - AsynchronousEvents

2016-12-01T19:35:42Z

Version 6.6.5 of The Server Framework was released today. This release is mainly a feature release with a few bug fixes.... Version 6.6.5 of The Server Framework was released today. This release is mainly a feature release with a few bug fixes. As always, see the release notes here, for full details of all changes. Bug fixes: Bug fix to JetByteTools::Socket::TAsyncSocket::ProcessAndGetNextOperation(). We now wrap the body of the function in an exception handler and abort the connection with JetByteTools::Socket::ConnectionClosureReason::FatalErrorAbort if an exception is thrown during processing. This fixes a bug whereby the connection would otherwise hang in these situations. Bug fix to JetByteTools::Win32::TReentrantLockableObjectTracksLockingThread to add a cast which is needed for some compilers. Bug fix to JetByteTools::WebSocket::HyBi::CProtocolHandler::HandleData() to remove an incorrect internal state validation exception which would generate spurious "No space left in read buffer." exceptions. Changes: Added JetByteTools::IO::IBuffer::OnBufferAddedToPool(), JetByteTools::IO::IBuffer::OnBufferRemovedFromPool(), JetByteTools::IO::IManageBufferLifeCycle::OnBufferAddedToPool() and JetByteTools::IO::IManageBufferLifeCycle::OnBufferRemovedFromPool(). These allow for correct management of dynamically allocated buffer data. Previously JetByteTools::IO::IMonitorBufferAllocation::OnBufferAllocated() was called whenever a buffer was allocated from the allocator OR a custom pool and JetByteTools::IO::IMonitorBufferAllocation::OnBufferReleased() was only called when the buffer was released to the allocator. This made it impossible to manage dynamically allocated buffer data that was created and destroyed by a derived allocator object using the monitoring interface to monitor the underlying allocator. Now JetByteTools::IO::IMonitorBufferAllocation::OnBufferReleased() is called when the buffer is added to a custom pool and this allows the monitor to match allocations and releases exactly. Added JetByteTools::IO::IAllocateBuffers::AllocateCustomSizedBuffer() which allows you to allocate a custom buffer from any allocator. The buffer will have the same user data slots as any other buffer allocated from the allocator but if it is larger than the allocator's buffer size the new buffer will NOT be pooled upon release. Added JETBYTE_ILLEGAL_BUFFER_USER_DATA_EXCEPTIONS which defaults to 0 and when set turns on range checking for user data indices in JetByteTools::IO::CBuffer and an index out of range exception is thrown if necessary. Added the ability to pass JUST a custom allocator into the constructor of JetByteTools::IO::CRotatingAsyncFileLog and allow the file log to manage the other resources that it needs directly. Previously you had to provide all of the resources or none. Added an override for JetByteTools::Socket::ISocketCallback::OnError() which takes a DWORD error code so that client code can selectively ignore errors by error code. Previously the only way to ignore errors was by the error message itself which is localised and therefore impossible to match reliably. Rationalised the status changes for JetByteTools::Socket::TStreamSocketServer<> and JetByteTools::Socket::TStreamSocketServerEx<> to remove some strangeness when shutdowns are initiated after the server has already shut down. Added new value to JetByteTools::Socket::ConnectionClosureReason, FatalErrorAbort. This is used if the framework itself needs to abort a connection for any reason. Added JetByteTools::Socket::CStreamSocketNamedConnectionCollection::GetCo[...]



Things have been busy! - Company News

2016-05-24T14:22:46Z

We've been busy with lots of custom development work over the last few months. Working with Eonic Gaming on their server for the Turf Battles Triumphus 3D MMORPG. Working with one of our security company clients on their video...
We've been busy with lots of custom development work over the last few months. Working with Eonic Gaming on their server for the Turf Battles Triumphus 3D MMORPG. Working with one of our security company clients on their video streaming server, adding RTP over TCP and HLS streams to the RTP over UDP of the original design. Improving the performance and functionality of the pluggable server that we wrote for one of our other gaming clients as they scale out into the cloud. And doing some consultancy for existing clients who want to expand their understanding and usage of The Server Framework.

The video streaming work was fun and went well and it was nice to have a well specified project using internet standards that we could really get our teeth into.

The work for Eonic Gaming has been especially interesting as we've worked closely with them to address their initial scalability issues and switch from a single server solution to a more traditional MMORPG style multiple server design.

We've also been working on several new releases of The Server Framework, one in the 6.6.x tree, a new, slimmed down 6.7 and some investigation into what could become massively modernised 7.0 release. But we'll see...

Now that things are finally becoming a little bit calmer our focus is on getting the 6.6.5 and 6.7 releases finalised and out the door.
(image)



TLS 1.2 handshake failure for certificates signed with MD5 - AsynchronousEvents

2016-06-25T10:59:19Z

A while back a client of mine had an issue with a TLS 1.2 connection failing during the TLS handshake. We couldn't see any issues with the code and if we only enabled TLS 1.1 on the server then the... A while back a client of mine had an issue with a TLS 1.2 connection failing during the TLS handshake. We couldn't see any issues with the code and if we only enabled TLS 1.1 on the server then the connection handshake worked just fine. Eventually we tracked the issue down to the fact that the certificate in use had been signed with MD5 and that MD5 isn't a valid hash algorithm for TLS 1.2 and so the handshake failed when SChannel attempted to validate the certificate and found that it was unacceptable. There's a Microsoft blog posting about this problem here. Annoyingly the error message for this is: "The client and server cannot communicate, because they do not possess a common algorithm." which is bang on the money, but doesn't help much until after you've worked out what the problem is. We were already dumping out the enabled algorithms and protocols and couldn't understand why the connection either wasn't working or wasn't downgrading to TLS 1.1. I've now added some certificate investigation code to the point where my example SChannel servers set up their contexts. This at least allows me to warn and degrade to TLS 1.1 if the certificate is not going to work with TLS 1.2. In production systems I expect we'd just fail to start up. CCredentials::Data data; data.cCreds = 1; data.paCred = &pCertContext; // Note that for TLS 1.2 you need a certificate that is signed with SHA1 and NOT MD5. // If you have an MD5 certificate then TLS 1.2 will not connect and will NOT downgrade // to TLS 1.1 or something else... DWORD dataSize = 0; if (::CertGetCertificateContextProperty( pCertContext, CERT_SIGN_HASH_CNG_ALG_PROP_ID, 0, &dataSize)) { ByteBuffer buffer(dataSize); BYTE *pData = buffer.GetBuffer(); if (::CertGetCertificateContextProperty( pCertContext, CERT_SIGN_HASH_CNG_ALG_PROP_ID, pData, &dataSize)) { const _tstring signHashDetails(reinterpret_cast(pData), dataSize); if (signHashDetails.find(_T("MD5")) != _tstring::npos) { if (enabledProtocols == 0 || enabledProtocols & SP_PROT_TLS1_2) { OutputEx(_T("Certificate is signed with: ") + signHashDetails); OutputEx(_T("MD5 hashes do not work with TLS 1.2 and connection downgrade does NOT occur")); OutputEx(_T("This results in failure to connect. Disabling TLS 1.2")); if (enabledProtocols) { enabledProtocols &= ~SP_PROT_TLS1_2; } if (!enabledProtocols) { // if enabledProtocols is zero then all protocols are // enabled, so we need to explicitly enable everything // except TLS1.2 enabledProtocols = SP_PROT_SSL3TLS1; } } } } } data.grbitEnabledProtocols = enabledProtocols; [...]



Latest release of The Server Framework: 6.6.4 - AsynchronousEvents

2016-12-01T19:36:24Z

Version 6.6.4 of The Server Framework was released today. This release is mainly a bug fix release for clients using WebSockets over SSL.... Version 6.6.4 of The Server Framework was released today. This release is mainly a bug fix release for clients using WebSockets over SSL. As always, see the release notes here, for full details of all changes. Bug fixes: Bug fix to JetByteTools::Win32::CallbackTimerQueueBase which prevents the timeout handle from ever being incremented to zero. It's unlikely but possible. Bug fix to JetByteTools::Win32::CBuffer and JetByteTools::Win32::CLockFreeBuffer to make code take notice of when m_maxBytesToRead is set. This was causing issues with WebSockets over SSL where the WebSocket code was setting a max read value and the SSL connector was ignoring it and providing more data than it should. Bug fix to JetByteTools::SSPI::SChannel::CAsyncConnector::PerformWrite() so that we deal with exceptions appropriately. Bug fix to JetByteTools::Socket::CFilterDataBase::WriteData() so that we deal with exceptions appropriately. Bug fix for a highly unlikely possible situation in JetByteTools::WebSocket::HyBi::CProtocolHandler::HandleData() where a buffer in the read request buffers queue contains no space for more data. Changes: Adjusted all includes to remove relative paths and instead be absolute in relation to the library root. So, rather than #include "..\Blah.h" it would always be #include "JetByteTools\Lib\Blah.h" this is due to a new warning in Visual Studio 2015 update 1. Added JetByteTools::Socket::ListenBacklogMaxwhich is a JetByteTools::Socket::ListenBacklog value that can be used to cause the listen queue for a socket to be set to SOMAXCONN. Added JetByteTools::Socket::TStreamSocketServerEx::SoMaxNumPendingAccepts which is the number of accepts that are posted if a listen backlog of JetByteTools::Socket::ListenBacklogMax is specified for the number of pending accepts. At present this is set to 200. Added JetByteTools::Socket::TStreamSocketServerEx::SetListeningParameters() which can be called after construction but before starting to accept connections and can specify the listen backlog and number of receives to post separately. This allows you to set the listen backlog to SOMAXCONN using JetByteTools::Socket::ListenBacklogMax and set the number of receives to something other than JetByteTools::Socket::TStreamSocketServerEx::SoMaxNumPendingAccepts. You can also set a flag which tells the server not to close the listening socket when the server reaches a connection limit, instead it will simply stop posting accepts and so new connections will get to wait in the listen queue until they time out or until the server posts a new accept request. Bug fix to to JetByteTools::Socket::TStreamSocketServerEx so that OnMaxConnections() is called correctly when the connection limit is reached. Previously it would often only be called the first time that the limit was reached. [...]



SChannel ALPN support, documentation "bug"... - Rambling Comments

2015-11-05T15:27:37Z

I'm looking at adding support for the TLS Application-Layer Protocol Negotiation Extension to The Server Framework as a precursor to looking into HTTP 2.0 support. This TLS extension is supported by both SChannel and OpenSSL. It's relatively new (only having been...

I'm looking at adding support for the TLS Application-Layer Protocol Negotiation Extension to The Server Framework as a precursor to looking into HTTP 2.0 support. This TLS extension is supported by both SChannel and OpenSSL. It's relatively new (only having been included in SChannel since June 2014 (!)) but there seems to be no documentation on MSDN for the structures and how to use them to add the appropriate entries to the TLS handshake and then to select and validate the protocol of your choice... Searching today with Google for something like "SEC_APPLICATION_PROTOCOL_NEGOTIATION_EXT" just gives just the usual selection of hits to people who think it's clever to run the Windows header files through their document generators and then publish the results on the web...

It's slightly easier with OpenSSL, but only because I can look at the source...

Anyway, long story short, I have the OpenSSL version working and am getting an HTTP 2.0 Connection Prefix from Chrome... I guess, for now, SChannel will have to wait.

I expect I'm being dense...

(image)



GetVersionInfoEx() deliberately broken in Windows 8.1/10 - Rambling Comments

2015-09-16T10:13:14Z

There should be a Developer Advocate position within Microsoft. They seriously need someone who doesn't "Drink the Kool-Aid" who can step in and tell the Program Managers or Marketing bods or whoever it is who makes all the really STUPID...

There should be a Developer Advocate position within Microsoft. They seriously need someone who doesn't "Drink the Kool-Aid" who can step in and tell the Program Managers or Marketing bods or whoever it is who makes all the really STUPID decisions that the decision is stupid and that they should get back in their box and leave the developers who support the Windows ecosystem by building software that runs on it to just get on with their lives...

Yes, this is going to be a rant...

So someone has decided that being able to query which version of the Operating System that your program is running on is a bad thing. If you try and call GetWindowsVersionEx() on Windows 8.1 or Windows 10 and you haven't specifically set your application's manifest to say that it's compatible with Windows 8.1 or Windows 10 then GetWindowsVersionEx() will lie to you and tell you that you're running on Windows 8.

Excuse me, but that's just brain dead. There's NO valid reason for doing this because you can't know WHY the caller needs this information.

OK, so it might be better, in some circumstances, to be checking for the presence of various function entry points in various dlls and dynamically working out that what you need is present BUT that's not the only reason that you might want to know what OS you're running on and it's not even the only GOOD reason.

My current situation is that Windows 10 spawns a conhost.exe process whenever you start a console process. The test code that I have for my Win32 Job API wrapper includes some checks to determine that the correct job accounting data is being returned from some test programs that it spawns. This accounting data is different under Windows 10 as there are more processes running within the job. Ideally I'd like to acknowledge that this is an OS difference, adjust the test to know that when it runs on this OS it needs to expect slightly different results and then move on with my life. Of course, as Microsoft has deliberately chosen to break the API that would let me cleanly work out what version I'm running on I have to spend time writing ranty blog postings and crufting up crappy work arounds.

(image)



Visual Studio 2015 Bugs - Save As existing file crash - Rambling Comments

2015-08-28T14:45:31Z

As I said a while ago, Visual Studio 2015 appears to have lots of pretty serious bugs. One that probably bites me more than many people is this Save As bug. If you have a project and you select a...

As I said a while ago, Visual Studio 2015 appears to have lots of pretty serious bugs.

One that probably bites me more than many people is this Save As bug. If you have a project and you select a file in that project and do a "Save As" such that the target file is also in the project then the VS2015 IDE crashes.

I tweeted about this a while back, and the Visual Studio Program Manager, Cathy Sullivan, wanted me to create a new project and see if the problem happened with it... Not quite sure why she couldn't do this herself, but anyway, here's a zip of a project created with VS2015 where there's a file called SaveMeAsTargetver.h. If you save that file as targetver.h then the IDE crashes.

(image)



Latest release of The Server Framework: 6.6.3 - AsynchronousEvents

2016-12-01T19:37:09Z

Version 6.6.3 of The Server Framework was released today. This release is mainly a bug fix release but it also adds support for Visual Studio 2015 and Windows 10 (though we don't explicitly use any Windows 10 APIs). There... Version 6.6.3 of The Server Framework was released today. This release is mainly a bug fix release but it also adds support for Visual Studio 2015 and Windows 10 (though we don't explicitly use any Windows 10 APIs). There are quite a lot of small changes due to us running Gimpel Lint over the code. Most of the changes will have fixed potential issues rather than issues that have actually been reported. However, there are two serious fixes for the SChannel Option Pack for an issue that was introduced in Release 6.6.2 and which could cause corruption of the data flow in some situations and stalling and CPU pegging in another. There is also a fix which may prevent UDP datagram corruption, See here for more details. As always, see the release notes here, for full details of all changes. All clients using Release 6.6.2 of the SChannel Option Pack are advised to upgrade to this release. Bug fixes: JetByteTools::IO::CBuffer::Clear() which now correctly sets m_ioSize to zero. JetByteTools::IO::CBufferList::BufferData so that it cannot access pBuffer before it's initialised. JetByteTools::IO::CLockableBufferProcessor::~CLockableBufferProcessor() to remove potential buffer leak. JetByteTools::IO::CAsyncFileReader where we were checking the validity of data response flags incorrectly. JetByteTools::SSPI::SChannel::CAsyncConnector::PerformWrite() where we could sometimes send invalid data. JetByteTools::SSPI::SChannel::CAsyncConnector where we could sometimes get stuck in an infinite loop if we had decrypted data available but no application level read pending. JetByteTools::Service::CService::ServiceMain() so that we set and clear the m_serviceStarting flag correctly. Fix to all WSASend() and WSARecv() calls to prevent multiple threads from calling into the API on the same socket at the same time. See here for more details. JetByteTools::Win32::CCallbackTimerWheel::EndTimeoutHandling() where we were leaking one shot timers. JetByteTools::Win32::CThreadPool::OnThreadPoolThreadStopped() and JetByteTools::Win32::CThreadPoolEx::OnThreadPoolThreadStopped() so that the call to the monitor's OnThreadPoolStopped() function is consistent and in the right place. JetByteTools::Win32::CSmartHandle to deal with closing pseudo thread or process handles. Fix to JetByteTools::Socket::CFilterDataBase to ensure that each filter knows how many writes it has issued and that they pass completions to the next layer when they should. JetByteTools::Win32::TReusableIdManager::InternalTryFree() so that we now correctly merge intervals. Note that this bug was purely in the way we stored the intervals (which was less efficient than it could be) and did not affect the function of the manager. JetByteTools::Win32::TZeroInitialiseExpandableBuffer::Resize() so that it doesn't trash memory if you reduce the size of the buffer. JetByteTools::Win32::TZeroInitialiseExpandableBuffer a fix to the assignment operator so that it actually compiles and works. Changes: Ran Visual Lint using Gimpel PC Lint on all code and adjusted to remove warnings and fix bugs. Added support for Visual Studio 2015. Removed all use of exception specifications. We only ever used throw() b[...]



Visual Studio 2015 RTM - I hope the first service release is soon... - Rambling Comments

2015-07-24T20:19:26Z

As usual I have used VMs to play with the CTP releases for the new version of Visual Studio and fixed up new warnings and build failures in my source so that I'm not too surprised when the RTM comes...

As usual I have used VMs to play with the CTP releases for the new version of Visual Studio and fixed up new warnings and build failures in my source so that I'm not too surprised when the RTM comes along.

Unfortunately as soon as I put VS2015 RTM on my build server and started running it as part of my CI system I started noticing strange things...

The first is that it seems to be possible to get VS2015 into a state where it will refuse to load C++ projects. I've no idea how I've managed this, but it's now happened twice. Rebooting the machine seems to fix things. The log file that the failure indicates might help shows that it's trying to load a the "Visual Studio C++ Project System Package" and it's getting an E_POINTER result...

The second issue seems to be a race condition in shutting down the new process that's used to help various parts of VS communicate with each other, VSHub.exe. My CI system will fire up a Visual Studio instance to build a project using /build and once that completes another project is likely to get built, sometimes two VS2015 builds will follow one another, other times there are other versions of Visual Studio building in between. VS 2015 appears to start an instance of VSHub.exe but only if one isn't running (or, more likely, it always starts one but if one's already running then the new ones shut down). Then when the last instance of VS quits VSHub.exe hangs around for a moment and then shuts down... It seems to crash if it's shutting down when a new instance of VS2015 is started (and, presumably decides that the VSHub.exe that is shutting down is the one to use). A fix for this issue is simply to keep VSHub.exe alive for the entire time that all VS2015 builds need to run and I can do this by having an instance of VS2015 open on the build machine whilst the CI system is running...

Thanks to Abe10, in this thread, for the workaround for the VSHub.exe issue.

6 out of 10 for this VS release Microsoft, try harder. I hope the Windows 10 team have better QA.

(image)



The real solution to "error MSB4175: The task factory "CodeTaskFactory" could not be loaded from the assembly" - Rambling Comments

2015-07-15T07:55:56Z

I now have a real solution to the problem that I outlined on Saturday night and this solution, unlike the one developed during my "Macallan driven development" session on Saturday evening actually works. The problem is that when using code... I now have a real solution to the problem that I outlined on Saturday night and this solution, unlike the one developed during my "Macallan driven development" session on Saturday evening actually works. The problem is that when using code to run build using VS2010 by running a project into devenv.com I get the following error message: C:\Program Files (x86)\MSBuild\Microsoft\VisualStudio\v12.0\CodeAnalysis\Microsoft.CodeAnalysis.targets(214,5): error MSB4175: The task factory "CodeTaskFactory" could not be loaded from the assembly "C:\Windows\Microsoft.NET\Framework\v4.0.30319\Microsoft.Build.Tasks.v12.0.dll". Could not load file or assembly 'file:///C:\Windows\Microsoft.NET\Framework\v4.0.30319\Microsoft.Build.Tasks.v12.0.dll' or one of its dependencies. The system cannot find the file specified. The project builds correctly from the command window using the exact same command line as I use in my calls to CreateProcess(). I was on the right track on Saturday evening. It is an environment issue, just not the one that I thought it was. I also now understand why I thought I'd solved the problem on Saturday night only to discover that the fix didn't work when I checked it again on Sunday morning. The problem is caused by the presence of "VisualStudioVersion" in the environment. This variable is set to 12.0 when my task runner program is run from within VS2013 either when debugging or running without the debugger attached but launching from within VS2013. The presence of this variable in the task runner's environment somehow causes VS2010 to use the latest version of MSBuild that ships with VS2013. When doing this MSBuild gets confused about which .Net framework directory to load assemblies from and fails to locate its dll. Removing the "VisualStudioVersion" from the inherited environment before launching VS2010 programatically fixes the problem. Running my task running from a normal command prompt, rather than from inside of VS2013, allows it to work correctly without needing to remove the variable (as the variable is only added by VS2013 itself). I expect that I tested like this on Saturday night and that's what made me think I'd solved the problem them. VS2013 actually adds several "VisualStudio" variables to the environment and I now remove all of them before running any task from my task runner. [...]



Solution to "error MSB4175: The task factory "CodeTaskFactory" could not be loaded from the assembly" - Rambling Comments

2015-07-12T07:47:58Z

Updated: 12 July 2015 - It seems that this is all incorrect... Upon running my tests again this morning I find that the x64 task runner also fails to run VS2010 correctly and so the environment differences are unlikely to... Updated: 12 July 2015 - It seems that this is all incorrect... Upon running my tests again this morning I find that the x64 task runner also fails to run VS2010 correctly and so the environment differences are unlikely to be the cause... So, having decided that my continuous integration system could be better, and having looked at JetBrains' TeamCity and decided that it doesn't quite fit I'm looking back at some code that I wrote back in 2008 when I last thought about this kind of thing... I have some code which works with trees of dependant tasks and triggers sub tasks once the tasks they depend on complete successfully. The code uses CreateProcess() and Win32 Jobs to control the tasks and it can run builds and tests using the various flavours of Visual Studio installed on the box in question. So far, so good as a starting point for an agent that can run multiple builds on a machine... My problem, well, one of them, being that when using VS2010 by running a project into devenv.com I get the following error message: C:\Program Files (x86)\MSBuild\Microsoft\VisualStudio\v12.0\CodeAnalysis\Microsoft.CodeAnalysis.targets(214,5): error MSB4175: The task factory "CodeTaskFactory" could not be loaded from the assembly "C:\Windows\Microsoft.NET\Framework\v4.0.30319\Microsoft.Build.Tasks.v12.0.dll". Could not load file or assembly 'file:///C:\Windows\Microsoft.NET\Framework\v4.0.30319\Microsoft.Build.Tasks.v12.0.dll' or one of its dependencies. The system cannot find the file specified. This is annoying, to say the least, VS2005, 2008 and 2012 and 2013 all work fine, it's just 2010 that fails. I googled the issue and the general consensus is that once you've installed VS2013 a different version of MSBuild is used. Of course that doesn't answer the question of why it 'works on my machine' except when my task runner is running the command... Eventually I got the task runner to dump the environment and did the same from the command prompt where the command worked. The difference being that the command prompt that worked had an 'x64' environment and the task runner had a different, 'x86/win32' environment... This was because the task runner was a Win32 process. Running the task runner as x64 fixed the issue. So there's probably a bug in the VS2013 installation code that fails to set some environment variables for the Win32 command prompt that are required once it's "screwed with" the existing MSBuild installation on the box... I'm expecting that manually adding the missing bits to the 'x86' environment will mean that the task runner will work as a Win32 build AND an x64 build... [...]



Bug hunting - Rambling Comments

2015-07-01T10:14:53Z

I've just spent a day tracking down a bug in a pending release of The Server Framework. It was an interesting, and actually quite enjoyable, journey but one that I shouldn't have had to make. The bug was due to... I've just spent a day tracking down a bug in a pending release of The Server Framework. It was an interesting, and actually quite enjoyable, journey but one that I shouldn't have had to make. The bug was due to a Windows API call being inserted between the previous API call and the call to GetLastError() to retrieve the error code on failure. The new API call overwrote the previous error value and this confused the error handling code for the previous API call. This was a bit of a "school boy error" and it was made worse by the fact that the code in question had a comment which clearly explained why nothing should be placed between the API call and the call to GetLastError(). So, what went wrong? This bug has severely delayed the release of 6.6.3 of The Server Framework. This is mainly due to the fact that it caused many of my "black-box" server tests to fail spectacularly with all manner of symptoms and with crash dumps indicating invalid memory accesses. This, in turn, caused several days of head scratching which used up the time I'd allocated to push the release out. Then client work took over and the release had to wait as I had no time slots large enough to allocate to finding and fixing the bug. Annoyingly all of the unit tests passed. It was only under stress that the code in question failed because the code path in question wasn't exercised in the unit tests. This is why I have the "black-box" server tests as part of my build and release process but they don't get run as often as they should as I don't practice continuous integration for various reasons. CI would have spotted the problem as soon as it was introduced into the codebase and it would have been easy to triage the changes, locate and fix the issue. The release in question has always been a collection of unimportant minor changes that has been allowed to accumulate issues over time and that had no real plan to it. All of the changes were being merged in from client fixes and so all of the fixes were active in the field to various degrees. It should have been easy to push the 'build, test and release' button at any time and push the release out. However when I did this the lack of CI meant that this was the first time that the code had been stress tested and, obviously, the clients who had the broken fix weren't putting the code into the same situation that my stress tests caused. The code change that caused the issue was in response to this issue relating to multiple threads calling WSASend() or WSARecv() at the same time. The solution was to add some per connection locking around the issuing of WSASend() and WSARecv() calls. Since there were several places in the code where this change was required it became a bit of a 'cut and paste' fix. The fix being applied to the TCP and UDP code in a similar pattern but in multiple places. One example of the change is this: const int result = ::WSARecv( socket.GetSocket(), buffer.GetWSABUF(), buffer.GetWSABUFCount(), &dwNumBytes, &dwFlags, NULL, // THIS makes us a sync read [...]



Hotfix now available for Slim Reader/Writer lock issue - Rambling Comments

2015-06-03T07:49:56Z

Back in September I mentioned that I had found a problem with my usage of Slim reader/writer locks. I expected this to be something that I was doing wrong but it turned out that it was a kernel bug. This...
Back in September I mentioned that I had found a problem with my usage of Slim reader/writer locks. I expected this to be something that I was doing wrong but it turned out that it was a kernel bug.

This morning Johan Torp tweeted that a hotfix for this issue is now available.

The note on the hotfix states: "Note These issues are not obvious and are difficult to debug. This update is intended for applicable systems to apply proactively in order to increase the reliability of these systems.", so go install it even if you don't know that you're having these problems right now...
(image)