Subscribe: Frans Bouma's blog
Added By: Feedage Forager Feedage Grade B rated
Language: English
data  database  entity  framework  it’s  llblgen pro  llblgen  model  net  new  pro  project  query  sql  support  work 
Rate this Feed
Rate this feedRate this feedRate this feedRate this feedRate this feed
Rate this feed 1 starRate this feed 2 starRate this feed 3 starRate this feed 4 starRate this feed 5 star

Comments (0)

Feed Details and Statistics Feed Statistics
Preview: Frans Bouma's blog

Frans Bouma's blog

The blog of Frans Bouma, creator and lead developer of LLBLGen Pro and ORM Profiler.


Let’s add a photo mode to Wolfenstein II: The New Colossus (PC)

Fri, 16 Feb 2018 13:43:04 GMT

Table of Contents Introduction Getting started, the tools we need What information do we need? The camera Game pause Hide HUD and other on-screen cruft To recap… Setting things up Memory allocation in windows processes Finding the FoV Finding the camera Hide HUD using the g_showhud variable Game pause / Timestop Reverse engineering and what it is all about Doom to the rescue “Dude, this is just plain luck” What else is there to find? Creating the camera tools Configuring IGCS to hook into the .exe (Raw)Input hooks Game exe hooks Setting up IGCS to write its camera data IGCS’ mainloop Testing the camera tools Determining hard-coded addresses directly from the exe’s bytes Compiling and testing the release build Conclusion Introduction I have a weird hobby: I like to take screenshots in games. Some call this activity game photography, and in a sense this is a bit like photography: composition, light, the things you have to deal with when taking photos in real life, are concepts you work with here as well. For photography you need a camera, and games in general present you with one, but there’s a problem: you aren’t going to be able to move and handle the camera however you please: it’s likely tied to the main character, there’s all kinds of info on the screen (Head Up Display or ‘HUD’) and the lens characteristics used are likely very wide (meaning: high Field of View (FoV) and low focal length). To solve this, we need to alter the game so we can move and change the game’s camera however we want, but how to do that? It’s not as if games in general offer options for that. In this article I hope to explain a bit how to create a DLL we’re going to inject into the game’s process to take over the camera and control other game aspects for the purpose of taking screenshots. For my screenshotting hobby I needed ways to control the cameras in the games I played and so I started another hobby: creating camera tools so one can take proper screenshots in games without limits. I do that using my own system, which you can find at the link, which is open source software (C++ with a little bit of x64 assembler). Sometimes a game offers a photo-mode which gives you control of the camera at times and lets you take screenshots within a given bounding box in the game. These are great if there’s nothing available, but in general they’re pretty limiting and the game we’re going to work with in this article doesn’t offer one so in cases where the photo-mode is pretty limited or non-existing, we have to add one ourselves. This all works only on PC, not consoles. On consoles, you have to work with the photo-modes the developer offers you. In case you’re wondering whether it’s legal: strictly speaking reverse engineering is allowed (Compaq vs IBM). Additionally we’re not changing game code on disk, but strictly in-memory at runtime. We’re not adding cheats at all, even though some of the tools we’re going to use are designed for these. But it’s a grey area, and to avoid getting banned online, I have the strict policy not to create camera tools for online only games, but only for off-line single-player games. The game we’re going to work with is the demo version of Wolfenstein 2: The New Colossus on PC. You can download it on Steam here. As the demo version is free, you can try everything discusses below yourself. The resulting tools also work on the full game, btw, so if you own the full game, you can use them there as well. Disclaimer: I’m not a 3D engine programmer, I develop an entity modeling and object-relational mapper system for .NET as my job, but in a past long forgotten I’ve spent over a decade in the Dutch demoscene (mostly on Amiga 500) and have written a system called ‘DemoGL’ back in the early 2000’s (a system to create OpenGL based demos with, before the shaders took over) so I’m familiar with assembler, C/C++ and graphics. If you’re not familiar with these, that’s fine, you’ll still likely pick up a few things here and there . Gett[...]

Optimizing memory usage

Mon, 04 Dec 2017 16:08:22 GMT

In this post I’ll discuss some memory optimization work I’ve done recently on the LLBLGen Pro runtime framework, v5.3.2. This is a popular (commercial) .NET ORM. LLBLGen Pro is on the market since 2003 and has seen a lot of refactoring work internally over the years, among them performance optimizations and memory usage optimizations. With new features come new challenges to make the overall framework perform the same or even faster and still perform the new features as well, so optimizing the core engine is a recurring effort. The optimizations discussed here form just a small subset of the massive amount of things you can do to optimize memory usage, but it could give some insights in how to get started and what to look for in your own code. As with all optimization work, it depends on the situation at hand and why that particular piece of code is less optimal so you can fix it properly. Before we can do any optimization however, we first have to define a couple of things so we don’t do any work  unnecessary or even harm the code we want to make more optimal. I won’t go into fine details about the differences between the various heaps, as that’s besides the scope of the article. A word of warning though: pre-mature optimization is bad. The things discussed here are solutions to problems I recognized. They weren’t applied because ‘they are faster, regardless’; they’re not rules you should live by. Sure, if you have the choice between two equal solutions, just pick the one which is more efficient. Performance optimization isn’t free: there’s always another thing you can optimize, and all it takes is a compromise elsewhere. Is that price worth paying? Up front you don’t know that, hence optimizing right from the start isn’t going to be beneficial in many cases: write clean, simple code that’s easy to maintain and understand. If there’s a slightly faster variant thinkable that’s way more complex and harder to understand, chances are it’s not worth it, and you only know after the fact, never before the fact: the code might be called just once, so optimizing it away is useless. Why optimize memory usage and when? .NET is a garbage-collector (GC) based platform and that brings great joys because you don’t have to care about memory allocation and management: the GC cleans up after you’ve trashed the room and in theory you never have to worry about any consequences. While trashing rooms as a wanna-be rock star might feel exhilarating, there’s no free lunch: not in hotels, nor inside your computer beneath the .NET CLR and its digital vacuum cleaner; something has to pick up the tab, and in the context of this article, it’s the GC. Allocating memory has a cost, albeit in general a small one: it takes a bit of time to allocate some memory, initialize it and make it look like that fancy StringBuilder you just new-ed up. When you add some strings to that StringBuilder and its internal buffer is too small, it has to allocate a new block of memory, copy over the original contents and free the old block. Those actions take time. Not ‘copy this large file to a usb-stick’-time, but doing it a lot of times will be noticeable. Besides the work the GC has to do for cleaning up the old buffer, the whole allocating/copying/freeing dance has another side effect: memory fragmentation. The GC/CLR compacts memory when it gets the chance. The main reason is that if your program needs to have a large block of continuous memory, chances are it’s available, and not fragmented over a million already deallocated buffers. In the latter case you’d get an out-of-memory exception as the block you’re requesting isn’t there. So in short: all that memory allocating work can be a serious burden for the overall performance of your software. The less memory your application allocates, the better: what isn’t allocated doesn’t need to be cleaned up / copied / packed and taken care of. The problem of course is: with a system where memory management is taken care of behi[...]

.NET (Micro)ORM fetch benchmark results and the fine details

Mon, 20 Nov 2017 15:41:30 GMT

For some time now I maintain the RawDataAccessBencher repository on github, and once in a while I run the suite of benchmarks and post the results. In the benchmarks I included all major micro-ORM and full ORM frameworks for .NET which have a significant group of users. The benchmarks are typical ‘micro’ benchmarks in that they run for a relatively short period of time. They also focus solely on fetch performance, how fast the given framework can create a query, fetch the resultset from the database server (on another machine on the network) and materialize objects from the resultset. In this article I’d like to present the results of the run of today. You can look at the results in a flat list here, or jump straight to the conclusions, by looking at the final results here. So how to read those values? The main point of this benchmark is to illustrate how fast a given framework is compared to its competition for set fetches and individual fetches. The numbers, while interesting, can’t be extrapolated to your own system. So if in the list of results framework X takes 300ms to fetch a set of rows, it doesn’t mean it will take 300ms on your hardware. It will only show it’s faster than framework Y which takes 400ms for the same data. It’s also likely X will be faster than Y on your system or any other system. I’ve updated RawDataAccessBencher recently with memory allocation values as well, so it’s now also more clear how much memory a framework consumes during a benchmark run. As some of you have asked me about graphs of these results, I’ve put the values in Excel and created some graphs for you to give a more visual overview who’s fast and who’s dropping the ball. I’ve combined the memory consumption graph with the timing graph so you can quickly see if a fast performance comes with a high price (lots of memory) or not. As data source, the Sales.SalesOrderHeader table is used from AdventureWorks. This is for a reason: it has fields with different types, has a couple of relationships with other entities, relationships of different kinds, has nullable fields and has a reasonable amount of data (31K rows). The benchmarks are divided in two groups: change-tracking fetches and non-change-tracking fetches. This is done mainly to avoid an apple-oranges comparison, as change-tracked fetches can take more performance than read-only, non-change tracked fetches. If you’re interested, please run the benchmark code on your own system. All dependencies are obtainable from NuGet, you only need the right database data/schema available in a SQL Server database, change the connection string and you’re good to go. The graphs With the graphs below, I’ve tried to explained why they are the way they are. All benchmarks were run on .NET 4.7, x64, release builds, clean network. Even though my own work is included in the benchmarks, I’ve tried to keep things fair. For all graphs: lower is better, the fastest framework / variant is placed at the left. Some frameworks have multiple results in one benchmark, e.g. because they offer multiple APIs or offer ways to pre-compile queries, which affect results. Each graph therefore is more than just the winner and the loser, ok? Good Ok, let’s dive in! Set fetches, non-change tracking This is a straight-forward fetch: fetch all rows in the table Sales.SalesOrderHeader into objects and don’t do any change tracking.   Here, the hand-written, hand-optimized materializer using a DbDataReader is the fastest. This is no surprise, there’s little room here to get any faster. On the other side of the spectrum we’re seeing Massive, which uses Expando/Dynamic to materialize the rows. I happen to be the maintainer of Massive and of course have profiled the codebase to see where the performance is lost. This came down to the caching mechanism implemented in the Expando implementation inside .NET. The only way to make this much faster was to implement my own Expando class. As these then wouldn’t be interchangeabl[...]

LLBLGen Pro v5.3 released!

Mon, 16 Oct 2017 07:48:55 GMT

Despite some large companies deciding they can go from preview to RTM in a week, I still think following the steps alpha-beta-RTM is better, so after the beta period of LLBLGen Pro v5.3, there’s now the RTM version of v5.3! For what’s new, you can read that in the beta announcement:

The version is free for everyone with a valid subscription.

Enjoy! (image)

LLBLGen Pro v5.3 Beta has been released!

Thu, 28 Sep 2017 14:49:47 GMT

We've released LLBLGen Pro v5.3 beta! Since the EAP we’ve added new functionality and tweaked some things too, based on feedback. Below is the full list of what’s new in v5.3 Beta, and this is also the list of new stuff we’ll include in v5.3 RTM, which is expected within a week or two. What's new in the LLBLGen Pro designer .NET Standard 2.0 support. The designer can now generate, for selected target platforms, .NET Standard 2.0 source code Firebird 3.0 support. LLBLGen Pro now supports Firebird 3.0, in both designer and LLBLGen Pro runtime. This effectively means that the new Boolean type is now fully recognized. The identity field support of Firebird 3.0 will be added as soon as the Firebird Client supports this. The latest releases RSS feed contents is now shown on the Hometab. This way you know when a new release is posted on the website. Default sequences for Postgresql in model first. When using model first with PostgreSql, the designer will for fields which have a sequence set in their field mappings set the sequence as the default sequence in the meta-data and will generate the sequence as the default value in DDL SQL scripts. This is needed for ORMs like Entity Framework 6 which don't have sequence support in their mappings and rely on default values for identity/sequence support. New Project setting: StoreTimeLastSyncIntoProject. A new project setting (under Entity model - General) has been added: StoreTimeLastSyncIntoProject which, when true (default) will make the designer store the last sync time with database or model with the catalog / schema. This setting can be switched off to e.g. avoid projects having merge conflicts based on time/date values of last-sync What's new in the LLBLGen Pro Runtime Framework New functionality / changes .NET Standard 2.0 support. The LLBLGen Pro Runtime Framework can now be used on .NET Standard 2.0 supporting platforms like .NET Core 2.0 and Xamarin. More information New RuntimeConfiguration system to configure LLBLGen Pro Runtime Framework specific behavior and settings in code instead of in .config files. Mainly meant for .NET Standard 2.0 using applications but it's also available for .NET Full.More information The Plain SQL API now has a FetchScalarQuery method to retrieve a scalar value from the DB using a plain SQL query. More information Query tagging. Linq and QuerySpec have the ability now to let you mark (tag) the SQL query or queries generated by the runtime with a tag. This tag is a string which is added as a comment before the actual query string. This way you can easily track back the SQL query on a database server with the actual calling code without a profiler / stacktrace. It's a low-impact way to make sure you can track down information at runtime in the future. More information: Linq, QuerySpec. Small changes The Runtime Framework is now compiled against .NET 4.5.2 QuerySpec: When a scalar query is executed using FetchScalar and T isn't a nullable type and the query returns a NULL value, the cast to T would result in a Null reference exception. We now added a check first which will throw an InvalidOperationException if a null value was returned and the type specified isn't a nullable type. Lambda caches and factory caches are now using a ConcurrentDictionary instead of a Dictionary per thread, which should save some memory in large applications. QuerySpec: an extension method has been added for ISortClause, DontEmitAliasForExpressionAggregateField, so the EmitAliasForExpressionAggregateField flag can be set using the fluent interface. See the LLBLGen Pro Runtime Framework Reference Manual for details. A new exception class has been added: ORMConfigurationException. This exception is thrown when an error is detected in the RuntimeConfiguration usage. Prefetch path fetches with a context are now using faster merges as the linear searches caused by the context usage in collections during merges are now no lo[...]

LLBLGen Pro v5.3 EAP Released!

Thu, 07 Sep 2017 08:56:01 GMT

After a failed attempt to port our LLBLGen Pro Runtime Framework to .NET Standard 1.6, we’ve ported it to .NET Standard 2.0 and managed to port over almost all features (A very small set of features aren’t supported but you either won’t miss them, or they’re out of our hands and limitations of .NET Core/standard)! This means finally there’s a full O/R mapper framework available for .NET Core 2.0 that’s fast, fully featured and ‘battle-tested’ for many years.

The port of our runtime framework comes with LLBLGen Pro v5.3, which we’ve released an Early Access Preview (EAP)version of yesterday. What’s also in v5.3? Well, Entity Framework Core 2.0 support. (image) The most notable additional features for EF Core 2.0 support from our perspective are entity splitting, valuetype definition support and if you are into it, VB.NET (image)

We’ve also changed/added some small things in the designer and runtime but these are relatively minor. We’ll be adding more features before RTM, which we expect to happen within a month or so.

I know a lot of people have been eagerly waiting .NET Core support for our runtime and I’m pleased to say it’s finally there. As the port is almost 100% feature compatible with the version for .NET Full, porting your own application to .NET Core should be relatively easy, or better: the ORM at least isn’t the problem here.

Stay tuned for more!

LLBLGen Pro v5.2 has been released! (and first quickstart video posted)

Thu, 04 May 2017 12:16:55 GMT

LLBLGen Pro v5.2 has been released! It’s a version with a couple of ‘highlight features’ like full model analysis and a lot of smaller new features / changes which together form a large list of new goodies, see my previous post on LLBLGen Pro v5.2.

We also started creating some videos, which are meant to get you started with LLBLGen Pro and show you how to get results very quickly, using various tools in the designer and the different ways people want to work (Model first, Database First). The first video is now done and it’s embedded below. It’s about getting from just a database to working sourcecode using database-first modeling. The next video will show how to get from scratch to a working database and working code using model-first modeling.


From Database To Code, using LLBLGen Pro

height="315" src="" frameborder="0" width="560" allowfullscreen>

LLBLGen Pro v5.2 Beta has been released!

Fri, 07 Apr 2017 09:03:10 GMT

Earlier this week we've released LLBLGen Pro v5.2 beta! See below for the new features and changes in this release. It's feature complete and comes with full documentation. We plan to release the RTM within 2 weeks or so. Initially we had planned for this release to port our LLBLGen Pro runtime framework to Netstandard1.6 (.net core) however we postponed that port to Netstandard2.0 and will release that port in v5.3. We did spend considerable time on the port to Netstandard 1.6 hence v5.2 took longer than expected. What's new in LLBLGen Pro v5.2 LLBLGen Pro Designer Major new features Model- and meta-data analysis. This analysis reports non-error issues and suggestions for the model and meta-data in the project in real time. Issues reported are e.g. duplicate FKs, an FK that's pointing to itself or e.g. if an FK field is shared among multiple FK constraints. Suggestions are e.g. how many meta-data elements don't have a mapping or that there are e.g. several PK-PK relationships in the model and they can be used to create inheritance hierarchies. All suggestions and warnings are reported with ready-to-use actions you can click on to directly execute an action (e.g. map all unmapped tables to entities). This system will be extended in the future with more analysis options. Several Entity Framework Core enhancements: v1.1.x is now supported, Firebird is now a supported database, read-only entities are now supported. .NETStandard 1.6 is now a supported platform, for Entity Framework Core and the DTO classes framework. Creating a project now asks for 'Initial contents' for the project. The options are: 'Empty project' (which is equal to the Create Project in previous versions), 'Relational model data retrieved from a database' or 'Relational model data container for a database'. The last two starts their respective wizards after the project has been created so the user doesn't have to look for the option in the user interface. Minor changes / fixes / new features The project file loader now reliably reports where in the file an xml error or other error was found during project load, and if possible which XML snippet caused it. This can greatly help finding errors in the project file, e.g. after a bad merge in a source control system. All drivers now construct the connectionstring with DbConnectionStringBuilder, which should provide more protection against badly formed connectionstrings due to userinput. A typeconverter which was moved to a different namespace is now normally loaded and used in existing projects which refer to the typeconverter with the old namespace name. The typename of the typeconverter itself has to be the same for this to make it work. Example: Foo.MyTC is used as typeconverter in a project. This typeconverter is moved to the namespace Bar: Bar.MyTC. Loading the project will now automatically use Bar.MyTC and not throw an error anymore that a type converter (Foo.MyTC) is missing. Various context menu additions for various nodes in the Project Explorer to make it easier to find how to reverse engineer project elements (e.g. Entities, Typed Views) from relational model data elements like Tables, Views. It will pick the first database in the project which has its sync source set to database, or if that's not available, set to 'mixed'. If none of the databases in the project have a sync source set to database or mixed, it will pick the first one. For more fine-grained control, the catalog explorer reverse engineer functionality is the best choice. QuickModel now has a flyout panel with quick help regarding commands so users can quickly check what the syntax is of common commands without first having to read a lot of documentation. It's opened by a new button in the QuickModel user interface called Command Help The settings Exclude orphaned element detected action and Non excludable orphaned element detected ac[...]

LLBLGen Pro v5.1 RTM has been released!

Tue, 22 Nov 2016 14:04:29 GMT

After 2 successful EAPs and a beta, we’re happy to release the RTM version of LLBLGen Pro v5.1! To see what’s new in detail please see the blogposts about EAP1, EAP2, and the Beta.

In short:

  • Entity Framework Core v1.x support. The LLBLGen Pro designer now supports Entity Framework Core v1.x, the new version of Microsoft's Entity Framework ORM. More information.
  • Plain SQL Api. The LLBLGen Pro Runtime Framework now lets you execute plain SQL statements using parameters directly onto the database, including fetching resultsets and projecting the resultsets to POCO classes. With this API you don't need a microORM on the side anymore for those situations where the runtime couldn't give you the right query. This API is also very fast: fetching a query is faster than most microORMs. More information.
  • Target hints. The LLBLGen Pro Runtime Framework now supports select target hints for SQL Server and MySQL in Linq and QuerySpec queries. More information.
  • Temporal (history) table support. The LLBLGen Pro Runtime Framework now supports temporal tables for SQL Server 2016 and higher/ DB2 , through Linq / QuerySpec (Select / fetch queries only). Temporal tables are an easy, friction free way to store previous versions of modified / deleted rows in a table. The temporal table support in LLBLGen Pro allows you to include older versions of entities when querying. More information.
  • Easier projections for QuerySpec and stored procedures. It's now easier to create projections of stored procedure resultsets onto POCO classes, all it takes is specifying the target type. More information.

The new runtime is of course lighting fast and beats most (micro)ORMs, including Dapper and Entity Framework Core. See the latest raw data-access benchmarks: See results from November 22nd using the latest RawDataAccessBenchmark code.

LLBLGen Pro v5.1 is a free upgrade for all v5 subscription owners.

LLBLGen Pro v5.1 Beta has been released!

Thu, 27 Oct 2016 09:23:00 GMT

LLBLGen Pro v5.1 beta is now available. It’s the last public build before RTM, which I hope will be within a week or two. Since EAP2 we’ve added the following features:

  • Entity Framework Core support in the designer
  • Newer versions of the UI controls in the designer for better HiDpi support
  • .NET 4.6.2 support for all supported ORM frameworks.

The headliner of course is EF Core support in the designer, which makes LLBLGen Pro the no.1. designer for Entity Framework Core projects. The EF Core support is for v1.0.1, and will be updated once Microsoft ships 1.1.0 RTM.

Entity Framework Core is supported on .NET Full and .NET Core, the generated code is generated with a csproj for .NET Full, but is easily converted to a .NET core project.

Almost there!

LLBLGen Pro v5.1 EAP 2 released!

Wed, 28 Sep 2016 12:50:13 GMT

Today we released the second ‘Early Access Program’ (EAP) build of LLBLGen Pro v5.1! Please see the previous post on the first EAP build and what’s included in that version. This version builds further on that foundation and includes new features (as well as all fixes made to EAP 1 and v5.0). Our EAP builds are ‘ready to release’ builds, with up to date documentation and are fully tested. Updated packages (marked ‘Alpha-20160928’) have been pushed to Nuget as well for the folks who prefer referencing nuget packages. What’s included in EAP 2? EAP 2 has the following new features, which are again all part of the LLBLGen Pro runtime framework. I’ll link to the documentation for details on the topics discussed below. Plain SQL API Using an ORM in general doesn’t confront you with plain SQL queries, often the APIs in ORMs don’t even allow you to specify a plain SQL query. This is OK for most situations but there are occasions where you might need to run a hand-optimized plain SQL query or have to execute a SQL query which can’t be generated by the ORM query API. Since the arrival of microORMs, which often have a plain SQL interface as their core query API, users of full ORM frameworks sometimes also use a microORM ‘on the side’ to perform these plain SQL queries when needed. While this might work, it also can be a bit of a problem as one can’t leverage aspects offered by the full ORM, like an active transaction or easy paging query generation. As we were already working on an addition to our POCO projection pipeline, we thought: why not open the runtime some more to be able to project plain SQL resultsets to POCOs as well? (and of course be able to execute SQL statements which don’t return a resultset). This is the new Plain SQL API. All methods have async/await variants of course. Executing a SQL statement Executing a SQL statement is meant to execute a non-resultset SQL query, e.g. to update or delete some rows. This is easy to do with the ExecuteSQL method. 1: Guid newId = Guid.Empty; 2: using(var adapter = new DataAccessAdapter()) 3: { 4: var q = @"SELECT @p0=NEWID();INSERT INTO MyTable (ID, Name) VALUES (@p0, @p1);"; 5: var idVar = new ParameterValue(ParameterDirection.InputOutput, dbType: DbType.Guid); 6: var result = adapter.ExecuteSQL(q, idVar, "NameValue"); 7: newId = (Guid)idVar.Value; 8: } The example above inserts a row into MyTable and returns the generated NEWID() Guid through an output parameter. It passes the value ‘NameValue’ as parameter value to the query as well. There’s a flexible way to specify parameters with just the values as arguments of ExecuteSQL and if needed, you can define a ParameterValue instance with parameter specifics, like direction, DbType, length, precision, scale etc. This is also the mechanism used to obtain the output parameter value after the query has been completed. Fetching a resultset Of course you can also fetch a resultset and project to POCO classes using the Plain SQL API. This is done using the FetchQuery method. See the following example: 1: List result = null; 2: using(var adapter = new DataAccessAdapter()) 3: { 4: result = adapter.FetchQuery( 5: "SELECT * FROM Customers WHERE Country IN (@p0, @p1) ORDER BY CustomerID DESC", 6: "USA", "Germany"); 7: } In the example above a query on the Customers table with a WHERE clause using two parameter values is projected onto the POCO class Customer. This is all nice and works great, but as the API is part of a full ORM, there’s more: we can leverage systems in the full framework. Here’s the same example again, but this time it utilizes the Resultset caching system in LLBLGen Pro and it also [...]

LLBLGen Pro v5.1 EAP1 released!

Tue, 30 Aug 2016 13:51:24 GMT

Today we released our first ‘Early Access Program’ build for LLBLGen Pro v5.1! When we moved to subscriptions (with perpetual licenses) when we released v5.0, the one thing I wanted to get rid of was the long delays between versions: no more 1.5-2 years of development to a massive release, but smaller releases which are given to the users quickly. So here is the first release of that. Additionally, we did a lot of work to make it release-ready. This means that the EAP build is a release like any other final release, including up-to-date documentation and fully tested. This is another big step for us, so we can switch an EAP build to ‘RTM’ at any time. In the coming months we’ll release more builds till we reach RTM. What’s included? This first EAP release contains the following features. We’re focusing on our state of the art ORM framework this time around for most of the features planned for v5.1 RTM. Temporal (history) table support (SQL Server 2016 / DB2 10) For select / fetch queries, the LLBLGen Pro runtime framework now supports temporal tables. A temporal table is a table with a coupled history table which is managed by the RDBMS: an update or delete of a row will copy the original row to the history table with two date markers to signal the period in which this row was valid. For more information about temporal tables in e.g. SQL Server, see this article. Temporal tables offer a transparent way to work with history data. This means you can now use temporal table predicates directly in Linq and QuerySpec (our fluent query API) queries to query on the current data but also on history data. See this example: 1: var q = from e in metaData.Employee 2: .ForSystemTime("BETWEEN {0} AND {1}", 3: fromDate, toDate) 4: where e.EmployeeId == 1 5: select e; Here a Linq query is defined which will query for the employee data of the employee with id ‘1’, and all rows valid between fromDate and toDate are included. This means that if the employee data of this particular employee was updated between these two dates, the original data which was updated will be included in the resultset as well. On IBM DB2, LLBLGen Pro also supports Business Time temporal table predicates, something which isn’t supported by SQL Server 2016. Table / View hints (SQL Server) and Index hints (MySQL) To specify a hint for the RDBMS query optimizer has been a requested feature for a long time, but I never found a proper way to make it easy to specify. With the temporal table support, the same mechanism can be used for specifying hints for table / views, in the case of SQL Server, and indexes, in the case of MySQL. All other databases which support hints (Oracle and DB2 come to mind) aren’t supported here, as they force the hints to be present as comments in the projection of the SELECT statement, and the hint system works with hints specified on elements in the FROM clause. This isn’t that bad however, as hints on Oracle and DB2 are heavily discouraged by the vendors of these databases so it’s unlikely we’ll add support for these particular hints later. To specify a table / view hint in a Linq query (or QuerySpec query), you simply call an extension method with the hint as argument as shown in the following example (SQL Server): 1: var q = from c in metaData.Customer 2: .WithHint("NOLOCK") 3: .WithHint("FORCESEEK") 4: join o in metaData.Order on c.CustomerId equals o.CustomerId 5: where o.EmployeeId > 4 6: select c; Here the target mapped by the entity ‘Customer’ will receive two hints specified in the SQL Query, nam[...]

The .NET support black hole

Thu, 23 Jun 2016 12:49:56 GMT

Today I ran into a bit of an issue. A work-item for LLBLGen Pro v5.1 is to support all the new features of SQL Server 2016. One of the features of SQL Server 2016 is ‘Always Encrypted’. You can enable this feature through the connection string, and after that all data-access is encrypted, no further coding needed. As this is a connection string setting, it’s enabled in every ORM out there out of the box, also in ours. That’s of course not the problem. The problem is adding more control over this feature to the developer writing code which targets SQL Server 2016. Starting in .NET 4.6, the SqlClient API offers a way to specify when and when not to encrypt using SQL Server 2016. This is done through a setting in SqlCommand’s constructor: you can specify a SqlCommandColumnEncryptionSetting value which gives you control over when to encrypt and when not to encrypt, which could greatly enhance performance if you just partly encrypt your catalog. There’s something odd though: Although SqlCommand has a property for this, ColumnEncryptionSetting, it’s read-only. Its backing variable is set through the constructor. Now, why is this a problem, you ask? Well, unless your code is creating SqlCommand instances directly, you can’t set the setting for a given SqlCommand instance: if you use DbProviderFactory, and most ORMs do, or if you use CreateCommand() on the connection object, you can’t set the setting. You can only set it if you directly use the constructor. Any database-generic code out there uses either DbProviderFactory or the CreateCommand() method on the connection object, and thus can’t use this feature. The problem Looking at this, somewhat terribly designed SqlCommand API, I wondered: “Ok, there’s a serious problem with this API, where can I give feedback so they can fix it?”. I couldn’t answer that. With a product you purchase from a vendor, you can do to that vendor’s support channel, ask them what they think should be done to get this resolved and you eventually reach a conclusion, but here, I have literally no idea. With .NET Core, most development regarding .NET within Microsoft is focused on that, and a lot of airtime is given to that on GitHub, blogposts etc. But .NET full, e.g. v4.6.2, where do you go with an issue like this? Connect? Mail someone within Microsoft, hoping they’ll route it to some person who won’t delete it right away and look at it? About Connect I’ll be short: no way in hell am I going to spent another second of my time on this planet in that crappy system. Not only does the .NET team not reply to any issues there, I still have some open issues there which are years old and no-one bothers to answer them. It’s like writing the problem into some text file and never look at it again, same result. About emailing someone within Microsoft: that might work, but it also might not. I happen to know some people within Microsoft and I’m sure they’ll at least read the email, but it’s a silly way to give feedback: here we have a mega-corporation which makes billions of dollars each month, says to be a developer focused company and you have to email your question to some employee and hope to get answers? How fucked up is that! Now, my technical issue with SqlCommand is not something everyone will run into. That’s also not the point. The point is: if there’s an issue with the .NET BCL/API, there should be a clear path to the teams working on this codebase to give them feedback, report issues and get things fixed. Today there’s none. Oh of course, if things related to .NET Core pop up, we can open an issue on GitHub and start the conversation there, but this isn’t related to .NET Core: SqlClient on .NET core doesn’t contain any encr[...]

LLBLGen Pro v5.0 RTM has been released!

Tue, 26 Apr 2016 13:32:38 GMT

After 1235 commits and 20 months of full time development, the work is done and we’ve released LLBLGen Pro v5.0! To see what’s new, please go to the official ‘New features’ page on our website, or read about them on this blog in the Beta announcement post.

I’m very happy with this release and how it turned out. I’m confident you’ll be too. (image) We have much more planned in the upcoming releases, so stay tuned!

To all the beta-testers, support team and others who have helped us to make it the best release we’ve ever done: Thank you!

Introducing DocNet, a static documentation site generator

Wed, 10 Feb 2016 14:27:48 GMT

Update: Since release, I’ve updated the repository with a github site with Docnet’s own documentation: It can be used as a showcase how things look in the output and also is now the main documentation site for Docnet. Also I’ve added Tabs support as a Markdown extension and fixes to the Theme! Original post: I wrote another generator! This time it’s for generating a static site for user documentation from markdown files. It’s called DocNet and it’s Open Source: It took me a good 7 days to build it. We at Solutions Design use it to create our user documentation for the upcoming LLBLGen Pro v5.0 release. The documentation for the designer is currently being worked on and gives a good overview how the output of DocNet looks like: LLBLGen Pro v5.0 designer documentation generated by DocNet (Work in progress!) The github repo contains a readme with all the options and how to use it, so I won’t mention it here again. It’s not complete yet, but it already can produce output we can live with. If you want to add features and work on this too, please feel free to provide a PR. “Why?” LLBLGen Pro exists now for 13 years. In those years the documentation has been shipped in .chm files and in recent years also as a static website generated from the same html files used to create the .chm file. If you’re not familiar with .chm files, they’re compacted archives, built with a compiler from Microsoft which hasn’t been updated in … 10 years? and use IE to render the pages (IE6’s render engine that is). In short, the format is completely outdated. Nowadays, documentation is shipped with HTML5 content, often online only, as a normal website. Additionally, the documentation is more and more written in Markdown. We wanted that too with our (massive) documentation for LLBLGen Pro. We had one extra requirement: the site should be equal locally as well as online, so people who have no internet connection at a given moment (airplane, train, dead wifi) can still browse the help locally. So in short: Documentation has to be written in markdown Documentation site is browsable locally and online with the same content Existing documentation files have to be convertible to markdown Has to use standard HTML5 / CSS / JS output Has to look modern. For our class library reference documentation I use Sandcastle Help File Builder, which is a brilliant piece of software. It works, output is known, no surprises, so that’s covered. The user documentation however, is another story. There are many, many different tools on the market to produce user documentation for you, in a lot of different formats. There are two categories: 1) a big IDE which manages content / topics / structure and lets you edit the content per topic in some form of editor and 2) a tool which consumes a set of files and produces a bunch of html files which form your user documentation. The tools in category 1) all do their job fairly well: some produce uglier output than the other, but in general they all are capable to produce user documentation that’s readable and usable. As we already have a massive amount of documentation in html files (which are hand-written in the html editor), the tools in this category didn’t really fit. This isn’t their fault, but simply our situation. The tools in category 2) are most of the time either in python, javascript or … php! Yes I was surprised too. The quality of these tools is often ‘ok’ but also need some work on your part. The tool which actually produced decent output was MkDocs. MkDocs is the tool that I had chosen at first but its local sit[...]

LLBLGen Pro v5.0 Beta has been released!

Mon, 01 Feb 2016 12:26:25 GMT

Since the first commit into the v5.0 fork back in the Fall 2014, we’ve been hard at work to make LLBLGen Pro v5.0 a worthy successor of the highly successful v4.x version. Today, we’ve released our hard work in beta, feature complete form: LLBLGen Pro v5.0 beta is now available.

For the features which are new, I’d like to refer to the two posts I did on the two CTPs we released: CTP1 features and CTP2 features, with one exception: the Home tab we added in CTP1 has been reworked into a different Home tab, as shown below.


The full change log since CTP2 can be found here.

Besides the ‘headliner features’ like the central relational model data synchronization and the derived model support for DTO and Document Databases, what I’m particularly pleased with is the tremendous amount of small things we managed to add: from simple things like defining a default database type for a .NET type when you do model-first development (so you can easily control if e.g. a .NET DateTime typed entity field maps to a Date DB type, Time DB type, or other) to things you’d expect like wiring up the references of multiple generated VS.NET projects automatically, or the 30%-40% faster linq/queryspec projection fetches in our runtime framework, making it faster than a lot of the well-known microORMs.

If you’re a v4.x licensee of LLBLGen Pro, the beta is available to you and can be downloaded from the ‘My Account’ page on our website.

Raw .NET Data Access / ORM Fetch benchmarks of 16-dec-2015

Wed, 16 Dec 2015 11:03:00 GMT

It’s been a while and I said before I wouldn’t post anything again regarding data-access benchmarks, but people have convinced me to continue with this as it has value and ignore the haters. So! Here we are. I expect you to read / know the disclaimer and understand what this benchmark is solely about (and thus also what it’s not about) in the post above. The RawDataAccessBencher code has been updated a couple of times since I posted the last time, and it’s more refined now, with better reporting, more ORMs and more features, like eager loading. The latest results can be found here. A couple of things of note, in random order: Entity Framework 7 RC1 (which we used here), is slow, but later builds are faster. It’s still not going to top any chart, but it’s currently faster than EF6, according to tests with a local build. We’ll update the benchmark with results from RC2 when it’s released. LLBLGen Pro v5.0, which is close to beta, has made a step forward with respect to performance, compared to the current version, v4.2. I’ve optimized in particular the non-change tracking projections as there was some room for improvement without cutting corners with respect to features. The results shown are achieved without generating any IL manually. The performance is better than I’d ever hoped to achieve, so I’m very pleased with the result. The NHibernate eager load results are likely sub-optimal, looking at the queries, however I couldn’t find a way to define a more optimal query in their (non-existing) docs. If someone has a way to create a more optimal query, please post a PR on GitHub The DNX build of the benchmark currently doesn’t seem to work, at least I can’t get it to start. This is likely due to the fact it was written for Beta8 and current bits are on RC1 and tooling changed a lot. As their tooling will change again before RTM, I’ll leave it at this for now and will look at it when DNX RTMs. The eager loading uses a 3-node graph: SalesOrderHeader (parent) and two related elements: Customer (m:1, so each SalesOrderHeader has one related Customer) and SalesOrderDetail (1:n). The graph is a graph with 2 edges which means frameworks using joins will run in a bit of a disadvantage, as the shortcoming of that approach is brought to light. The eager load benchmark fetches 1000 parents.  The eager loading only benches change tracking fetches and only on full ORMs. I am aware that e.g. Dapper has a feature to materialize related elements using a joined set, however it would require pre-defining the query on the related elements, which is actually a job the ORM should do, hence I decided not to do this for now. Perhaps in the future.  The new speed king seems to be Linq to DB, it’s very close to the hand-written materializer, which is a big achievement. I have no idea how it stacks up against the other micros in terms of features however. (Update) I almost forgot to show an interesting graph, which is taken with dotMemory profiler from Jetbrains during a separate run of the benchmarks (so not the one taking the results as profiling slows things down). It clearly shows Entity Framework 7 RC1 has a serious memory leak: (update) As some people can’t view pastebin links, I’ve included all results (also from the past) as local files to the github repository. [...]

What’s new in LLBLGen Pro v5.0 CTP 2

Fri, 30 Oct 2015 11:01:55 GMT

We’ve released the second CTP for LLBLGen Pro v5.0! Since the first CTP which was released back in March, we’ve been hard at work to implement features we wanted for v5.0. It’s taken a bit longer than expected as the main feature, Derived Models (more on that below), turned out to be a bigger feature than we initially thought and it affected more aspects of the designer than anticipated at first. Nevertheless, I’m very happy with the result, as it turned out even better than I imagined. This CTP is the last one before beta and RTM. It’s open for all V4 licensees and comes with its own temporary license which expires on December 31st, 2015. Beta and RTM are expected in Q1 2016. To download, log into our website and go to ‘My Account->Downloads->4.2->Betas’. So what’s new since CTP 1? I’ve described the highlights below. Of course the usual pack of bug fixes and improvements across the board are also included in this CTP. Derived Models Ever since v4 I wanted to do more with the designer, to do more with the entity model a user defines using it: we have all this model information contained in a single place and there should be more ways to leverage that than just for a mapping model to use with an ORM. The idea started when I looked into usage patterns again where ORM models are used. Nowadays most people define a different model in their application which is used to transport data from the ORM to the UI, e.g. across service boundaries or inside an MVC model class. This model is often hand-written, and tied to the entity classes using hand-written queries or Automapper mapping files. In other words: a hand-written chain of dependencies stored in multiple places which can break at any moment without noticing. Another use case of models defined on top of an entity model is with document databases. Web applications nowadays aren’t just monolithic stacks with one database and one runtime, they’re built as a group of technologies working together in harmony: a UI on the client in JS, a backend targeting multiple databases using multiple paradigms, etc. It’s more and more common to store non-volatile data in de-normalized form inside a document database for faster querying, and update that data on a regular basis, at runtime. This data is e.g. retrieved from the relational database which is used with the ORM and entity classes. For these two scenarios I’ve developed the Derived Models feature: Derived Models are models of hierarchical elements defined on top of the entity model. They define an optionally de-normalized form of the underlying abstract entity model. This all might sound complicated, but it’s actually very straightforward. Let’s see a picture of the editor to get started. (click the picture below for a bigger version) For the picture, I’ve picked a complicated scenario as it illustrates most of what you can do with the editor. On the far left of the designer you see the Project Explorer which is the overview of the elements in your project. The project loaded has an entity model with entities in various inheritance hierarchies, and a Derived Model called Docs. The Derived Model uses the target framework Document Database. We’ll get to that in a minute. The editor shown in the screenshot shows the derived element BoardMember, which is derived from the entity BoardMember. On the left of the editor you see the entity hierarchy in tree form: you can navigate from the root entity BoardMember to related entities infinitely. On the right you’ll see the shape of the derived element BoardMember. When you check a chec[...]

LLBLGen Pro v5.0 CTP 1 released!

Tue, 17 Mar 2015 15:20:11 GMT

We’ve released LLBLGen Pro  v5.0 CTP 1! It’s a Community Technical Preview (CTP) of the upcoming v5.0 which is in development since fall 2014. The CTP is open for all v4.x customers (it’s in the customer area, in the v4.2, betas section) and comes with a time-limited license which expires on June 1st, 2015. As this isn’t a full beta, (much) more features are added before beta hits. Below I’d like to show some of the new features in action. Click the screenshots for a bigger version. New, skinnable UI A new UI was long overdue. The current (v4.2) version still uses VS.NET 2008 like toolbars/controls and it looks just… dated. So, we ported the complete UI to DevExpress controls (still winforms though) as some of our controls were already based on those. The screenshots below show the default VS.NET 2013 white theme. New Relational Model Data Sync System In short: the Sync system. Sync replaces both database first related actions like refresh catalog and model first related actions like auto-map and adjust relational model data. It allows a sync source to be set for a schema which controls the source from where table related relational model data is obtained from: the database or the entity model. Stored procedures/views/tvfs are always obtained from the database. Everything is managed from a single tab, the Sync Relational Model Data tab, which is opened by clicking the sync button on the toolbar or menu item in the project menu. A big benefit from the new system is that it will function even when the project contains errors: it's no longer necessary to correct project elements before a refresh. It also doesn't adjust relational model data on database synced schemas, so it's no longer required to export DDL SQL before code generation because the validation adjusted some fields based on a change. Revamped Home tab The Home tab now shows active tiles the user can click to navigate through one of the scenarios (database first / model first and new / existing project). Which tiles are shown depends on what the user did last and the state of the designer. It's an easy way to get started with the designer and replaces the webpage based home tab which was static. It acts like a wizard in disguise: you can do the basic tasks right from the home tab, by clicking tiles. Here I’ve opened a Database first project and the state of the designer now shows different tiles with different actions. Search in Project Explorer / Catalog Explorer Directly available in the Project Explorer and Catalog Explorer are the search boxes: type in any string and the nodes matching the string are kept, all other nodes are filtered out. This allows finding elements with ease. Removing the search replaces the tree as it was before the search. In the below screenshot I’ve searched for ‘Sales’ in both the project explorer and the catalog explorer. It shows all nodes matching the string specified, including parent nodes (to give context) and hiding other nodes. The more advanced, LINQ query based search is still available in the designer, but this dedicated search on the two explorer panes is easier to use and requires no extra forms to navigate. Real Time Validation The designer got a real time system to schedule and run tasks at will through several dedicated dispatch queues. This greatly helps offload work from the UI thread while not having to mess with multi-threading as it utilizes the .NET 4.5 Task Parallel Library. Configuring work is as easy as defining an event, a handler and which dispatch queue to run the handler call and the syste[...]

LLBLGen Pro Runtime Libraries and ORM Profiler interceptors are now available on nuget

Tue, 10 Feb 2015 16:33:58 GMT

I caved. For years I’ve denied requests from customers to publish the LLBLGen Pro runtime framework assemblies on nuget, for the reason that if we had to introduce an emergency fix in the runtimes which also required template changes, people with dependencies on the nuget packages would have a problem. While this might be true in theory, in practice it’s so uncommon that this will happen, it more and more turned into an excuse. Add to that that customers started publishing the runtimes themselves on nuget, it was time to bite the bullet and publish the runtimes ourselves, officially. So we did. At the same time we published the interceptor assemblies of ORM Profiler on nuget. The URLs For LLBLGen Pro: For ORM Profiler: Who are these assemblies for? The assemblies are for customers who want to stay up to date with the latest runtimes. Every time we publish a new build, the runtimes and interceptor dlls are automatically updated with the latest build. We never introduce breaking changes in released assemblies, so they’re safe to use in code and update to the latest version. How are they versioned? The LLBLGen Pro Runtime Framework assemblies are versioned as: 4.2.yyyymmdd, where yyymmdd is the build-date. The ORM Profiler interceptors are versioned as: 1.5.yyyymmdd. What’s in the packages? The DQE packages come with a single DLL, the DQE dll, and have a dependency on the ORMSupportClasses package. The ORMSupportClasses package contains both the .NET 3.5 build and the .NET 4.5 build with async support: if your project targets .NET 4.5 you automatically will reference the .NET 4.5 build with async support. The Interceptor packages contain the interceptor dll and support dlls which don’t need their separate package. The Entity Framework interceptor has a dependency on Entity Framework 6. Do the DQE packages depend on ADO.NET provider packages? No, as all DQEs work with the DbProviderFactory system and don’t need the using project to reference an ADO.NET provider reference. The ADO.NET provider has to be present on the system, but as the provider assembly doesn’t need to be referenced by the VS.NET project the DQE package doesn’t need a direct dependency on the related ADO.NET provider package as that would mean the ADO.NET provider dll would be directly referenced after the DQE package has been installed. Hope this helps the customers out who have asked us for so long for this feature [...]