Subscribe: Coding the Architecture - sbrown
http://www.codingthearchitecture.com/authors/sbrown/rss.xml
Preview: Coding the Architecture - sbrown

Coding the Architecture - sbrown



Software architecture for developers



Last Build Date: Wed, 11 Oct 2017 08:56:02 GMT

Copyright: Coding the Architecture
 



Evolutionary design still requires up front thinking

Wed, 11 Oct 2017 08:56:02 GMT

There's a great post by Joshua Kerievsky titled The Day We Stopped Sprinting that itself references an older post called Evolutionary Design, which talks about the need to create a "primitive whole" initially, before iterating to improve it over time. The important thing to note here is that the primitive whole isn't just a bunch of unassembled parts, it's a "under-developed whole" and something that does work (to some extent, anyway). The example used in the illustration is a guitar, with the first version being something that looks like a very primitive guitar. Something that's often missed here, or not discussed explicitly, is that you still need to do some up front thinking in order to get to that primitive whole, and to create an initial vision or starting point. After all, the primitive guitar is still a guitar, which implies that somebody somewhere had a vision early on that a guitar was the thing that was needed. In my experience, I've seen many teams misinterpret "evolutionary design" and "emergent architecture" to mean, "you don't need to do any design up front at all". As Dave Thomas says, "big design up front is dumb, but doing no design up front is even dumber". During my software architecture workshops, I occasionally see teams draw nothing more than a visual summary of the requirements when asked to work in groups and design a software solution. The requirements are based upon a "financial risk system" for a bank, and sometimes groups will literally just draw a box labelled "Financial Risk System" before proudly proclaiming, "that's all of the up front design we need to do, we're agile". Yes, this really has, and still does happen from time to time. Engaging with the problem Many years ago, my boss gave me a small internal software project to work on. I don't remember the exact details, but he basically gave me a problem statement and told me to design a software solution. After a couple of hours, I presented my solution. It was instantly ripped apart and he told me that I hadn't properly engaged with the problem. He was right. I'd presented a very simple solution that didn't cater for any of the complexity of the problem space, mainly because I hadn't actually uncovered it yet. On my travels around the world, I regularly see the same thing, and people really struggle with the concept of doing up front design. Many of them also never really engage their minds in the problem, evidenced by one or more diagrams that present a very simplified and superficial view of the solution, such as the typical "logical view" diagrams below. Whenever I'm doing an up front design exercise, I want it to be quick and efficient while still providing as much value as possible. The process of doing some up front design provides you with a way to engage with the problem space and create a starting point for your vision of the thing you want to build. For me, doing up front design is about understanding the structure of the thing you're going to build, creating a starting point and vision for the team to work with and identifying/mitigating the highest priority risks. As I've discovered, a simple way to make this happen is to encourage people to actually produce some lightweight yet structured artifacts, like the diagrams in my C4 model, as a part of the design process. When they do this, conversations change because it forces people to engage with the problem. Up front design is therefore a crucial part of a longer evolutionary design process. Without it, you're just lost. [...]



PaaS for Java developers - Part 4

Tue, 10 Oct 2017 13:24:36 GMT

Previous parts of this blog post series have provided an overview of Cloud Foundry from a number of different perspectives; including the high-level concepts, vendor lock-in and the Pivotal Web Services marketplace services. In this part, we'll look at how Cloud Foundry makes it trivial to perform zero-downtime deployments. Blue-Green Deployments As a quick introduction to this topic, imagine that you have a Java web application running somewhere. A simple way to upgrade that application to a new version is to stop the application, update the relevant deployment artifacts (e.g. a .JAR or .WAR file), and then restart it. Some web application servers provide support for hot-swapping applications, but the principle is the same. Although this works, users of your application are likely to encounter downtime because the application will be unavailable for a short period of time. Over the years, we've created a number of techniques to deal with this issue, one of the most popular being Blue-Green Deployments, where a (physical or virtual) router is used to switch traffic from one running instance of your application to another. Although this might sound like an advanced technique, tools like Cloud Foundry make this feasible for teams of any size to achieve. As we saw in previous blog posts, Structurizr consists of two Java/Spring web applications; a "Web Application" (serving HTML, CSS and JavaScript) and an "API Application" (allowing clients to GET or PUT software architecture workspaces). The build and deployment process is fully automated, triggered by TeamCity running on an Amazon EC2 server waiting for commits to the git repository. In summary, this build and deployment process performs the following steps: Resolve dependencies. Initialise build directories, increment build number, etc. Compile code (production and tests). Run unit/class tests. Run integration/component tests. Create deployment artifacts (e.g. .WAR files). Push the API Application to Pivotal Web Services. Run e2e/system tests on the API Application. Push the Web Application to Pivotal Web Services. Run e2e/system tests on the Web Application. Make the API Application live. Make the Web Application live. Generate and publish new software architecture diagrams and documentation. Push applications to Pivotal Web Services Assuming that the build and tests were successful, the build process will push each of the API and Web Applications to Pivotal Web Services. The Cloud Foundry command line interface is installed on the build server, and the build script simply uses the "cf push" command to push the .WAR files. The "--no-start" flag is used so that the application is pushed, but not started, and this is done so that application environment variables (e.g. configuration) can be set using the "cf env" command. Once the configuration has been set, the "cf scale" command is used to set the desired number of instances and RAM, before actually starting the application. At this point, the applications are running but only accessible using a temporary URL that includes the build number (e.g. "https://structurizr-web-123.cfapps.io"). With the applications running, the build script can now run a series of end-to-end tests (a mixture of "smoke tests" and system tests), in order to verify that the new versions of the applications are running as expected. These tests include scenarios such as signing in, getting/putting software architecture models, etc. Making the new versions live If the end-to-end tests pass, the next step is to make these new versions of the applications live. This involves using the Cloud Foundry command line interface to map the live URL to the new versions of the applications ("cf map-route"), while removing it from the old versions ("cf unmap-route"). This process makes use of the Cloud Foundry router, which allows you to configure the URLs that are used to access running applications. If everything is successful, finally[...]



PaaS for Java developers - Part 3

Fri, 29 Sep 2017 16:16:43 GMT

Marketplace services I want to start part 3 by saying that I really do like and recommend Pivotal Web Services and Cloud Foundry as a simple and robust way to deploy Java applications. I've been running Structurizr on Pivotal Web Services for over 3 years now and I've had very few issues with the core platform. The marketplace services, on the other, are a different story. In addition to providing a deployment platform to run your code, most of the Platform as a Service providers (Pivotal Web Services, Heroku, Azure, etc) provide a collection of "marketplace services". These are essentially add-on services that give you easy access to databases, messaging providers, monitoring tools, etc. As I write this, the Pivotal Web Services marketplace includes many of the popular technologies you would expect to see; including MySQL, PostgreSQL, Redis, Memcached, MongoDB, RabbitMQ, etc. MySQL as a service Let's imagine that you're building a Java web application and you'd like to store data in a MySQL database. You have a few options. One option is to build your own database server somewhere like Amazon AWS. Of course, you need to have the skills to do this and, given that part 1 was all about the benefits of PaaS over building your own infrastructure, the DIY approach is not necessarily appealing for everybody. Another option is to find a "Database as a Service" provider that will create and run a MySQL server for you. ClearDB is one such example, and it's also available on the Pivotal Web Services marketplace. All you need to do is create a subscription to ClearDB through the marketplace (there is a free plan), connect to the database and create your schema. That's it. Most of the operational aspects of the MySQL database are taken care of; including backups and replication. To connect your Java application to ClearDB, again, you have some options. The first is to place the database endpoint URL, username and password in configuration, like you might normally do. The other option is to use the Cloud Foundry command line interface to issue a "cf bind" command to bind your ClearDB database instance to your application instance(s), and use Cloud Foundry's auto-reconfiguration feature. If you're building a Spring-based application and you have a MySQL DataSource configured (some caveats apply), Cloud Foundry will automagically reconfigure the DataSource to point to the MySQL database that you have bound to your application. When you're getting started, this is a fantastic feature as it's one less thing to worry about. It also means that you don't need to update URLs, usernames and passwords if they change. I used this approach for a couple of years and, if you look at the Structurizr changelog, you can see the build number isn't far off 1000. Each build number represents a separate (automated) deployment to Pivotal Web Services. So I've run a lot of builds. And most of them have worked. Occasionally though, I would see deployments fail because services (like ClearDB) couldn't be bound to my application instances. Often these were transient errors, and restarting the deployment process would fix it. Other times I had to raise a support ticket because there was literally nothing I could do. One of the big problems with PaaS is that you're stuck when it goes wrong, because you don't have access to the underlying infrastructure. Thankfully this didn't happen often enough to cause me any real concern, but it was annoying nonetheless. More annoying was a little bug that I found with Structurizr and UTF-8 character encoding. When people sign up for an account, a record is stored in MySQL and a "please verify your e-mail address" e-mail is sent. If the person's name included any UTF-8 characters, it would look fine in the initial e-mail but not in subsequent e-mails. The problem was that the UTF-8 characters were not being stored correctly in MySQL. After replicating the problem in my dev environment, I was able to fix it by adding a c[...]



PaaS for Java developers - Part 2

Thu, 28 Sep 2017 15:21:29 GMT

Vendor lock-in? In part 1, I introduced Platform as a Service (PaaS) and discussed how you can use Pivotal Web Services and Cloud Foundry as a way to easily deploy applications without worrying about the underlying infrastructure. A common misconception with all of this is that using Cloud Foundry (and Pivotal Web Services, or another implementation) results in vendor lock-in. Back to Structurizr, which is a collection of tooling to visualise and document software architecture. The system context diagram looks like this: In summary, authenticated users create and upload software architecture models using the Structurizr Client libraries (Java and .NET), and then view the content of those models via the web. Structurizr uses SendGrid to send e-mails, and all payment processing is performed by a combination of Taxamo and Braintree Payments. Some other services (e.g. CloudFlare, Pingdom and Papertrail) are also used, but not shown on the diagram. From a (C4 model) containers perspective, Structurizr is as follows (the external services have been omitted from the diagram because they are not relevant to this discussion): In essence, Structurizr is made up of a client application running in the web browser (HTML, CSS and JavaScript), while the server-side consists of a Java web application serving https://structurizr.com, another serving https://api.structurizr.com plus some data stores (MySQL, Redis and Amazon S3). The two Java web applications are running on Pivotal Web Services. Both of the Java web applications are based upon Spring MVC and they are implemented following a number of the principles described in the twelve-factor methodology. In reality though, from a technical perspective, both applications are just typical Java web applications that have been designed to run on the cloud. Both applications are stateless and they don't write important information to the local file system. Vendor lock-in and migration cost Let's talk about vendor lock-in or, as Sam Newman says, "don't think lock-in, think migration cost". All development for Structurizr is done using IntelliJ IDEA on a Mac, with Vagrant being used for running local copies of MySQL and Redis. Nothing in the codebase is tied to, or even aware of, Cloud Foundry and I don't have Cloud Foundry running locally. The Java applications are standard Java EE .WAR files that are deployed to an Apache Tomcat instance running locally. Pushing the Structurizr "Web Application" (using the "cf push" command) results in the web application being deployed on Pivotal Web Services and available at a URL of https://structurizr-web.cfapps.io. If I wanted to migrate that web application to another provider, here's what I would need to do. I would first need to find another PaaS that supports Java 8 and Apache Tomcat 8.x, or build my own server to do this. The JVM additionally requires that the "Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files" are installed, because the web application is making use of some of the stronger encryption algorithms when storing data. Given that there are multiple instances of the web application running (Pivotal Web Services transparently handles this behind a single URL), I need to mirror this setup too. The DNS for the structurizr.com domain is being managed by CloudFlare, and some CNAME records need to be changed to reflect the new deployment location. That's it. My deployment script will need to change, but no code changes are required. Pivotal Web Services does provide some additional features on top of Cloud Foundry, such as the their dashboards, which are handy for monitoring and management, but they are not an essential part of my applications. In summary, I don't really have any vendor lock-in and the migration cost is low. After all, my Java web applications are just regular Java web applications, with no dependencie[...]



PaaS for Java developers - Part 1

Wed, 27 Sep 2017 19:22:45 GMT

Introduction I've been a software developer for over 20 years, during which time I've built many different types of software systems in many different environments, the majority centred around Java and web technologies, but I've used others too. I've also been fortunate enough to have been involved in the full life cycle of software development; from inception through to delivery. What I haven't done much of is infrastructure. Sure, I've spun up a few servers but the majority of production infrastructure provisioning was performed by an infrastructure team. I don't think this is unusual. Provisioning production-grade infrastructure is a specialised task; requiring knowledge about installing, configuring and hardening operating systems, databases, load balancers, firewalls, etc. If I'm honest, it's not something I'm particularly interested in either, especially given the never-ending supply of things to learn in the software space. I remember having a number of discussions with people at Craft Conference in Budapest, three years ago, discussing deployment options for my startup called Structurizr. An early demo was running on a single Ubuntu server at Rackspace, but this raised a number of infrastructure and operations questions that I didn't necessarily have the knowledge to answer. One of the suggestions I received was that I should look at Pivotal Web Services. I'd played with this during a pilot project to run programming clubs at some of the local schools here in Jersey, but I'd not considered it for actually running my own apps. Pivotal Web Services and Cloud Foundry Pivotal Web Services is a commercial service offering based upon Cloud Foundry, which provides a "Platform as a Service" (PaaS). In essence, Cloud Foundry provides an application deployment platform via an API, abstracting the underlying infrastructure, whether installed on a public cloud, private cloud or a bunch of servers in a datacenter. Imagine that you're building a Java web application to run on Apache Tomcat. To get this running on the Internet, you need to provision a server somewhere, install Java and install Apache Tomcat before you can deploy your application. You also need to harden the server, configure SSL, apply patches on a regular basis, etc. And if you wanted two instances for better resilience, you now need to spin up a second server, repeat the previous steps and configure both servers to sit behind a load balancer (for example). Of course, you could, and probably would, use Vagrant, Chef, Puppet, Docker, etc to automate most of these steps. With the Cloud Foundry command line interface installed, if I have a deployable Java .WAR or .JAR file, I can simply run a "cf push" command to deploy it to Pivotal Web Services. That's it! I can also use the "cf scale" command to ask Pivotal Web Services to scale up the application (e.g. add more RAM), or scale out the application (e.g. add more instances). If I need to update the application, another "cf push" command is all it takes. Doing this will stop the currently running instance and replace it with the new version, so some downtime will be experienced. However, Cloud Foundry makes it super-easy to do Blue-Green deployments using the command line interface. As a Java developer, this gives you the ability to setup a zero-downtime continuous delivery pipeline in minutes, without any specialised knowledge of the underlying infrastructure. Applications over infrastructure As Joshua McKenty says on a recent podcast with Cisco Cloud, Pivotal Web Services and Cloud Foundry provides a higher level of abstraction for developers to work with. As a software developer, I'm dealing with applications, rather than infrastructure or containers. There's a lot of hype around Docker at the moment. And I think Docker is a fantastic technology. But, as a software developer, I want to deal with applications, not containers and infrastructure. I think[...]



Free e-books for software architecture meetups

Tue, 29 Aug 2017 10:46:00 GMT

To get more people thinking and talking about software architecture, I'm offering free copies of my Software Architecture for Developers ebooks for meetups. Simply organise a meetup on a software architecture related topic (see below) and send me a link to your meetup/event page by e-mail (simon.brown@codingthearchitecture.com). I will help you promote the event on Twitter, etc. Then, a few days before the meetup itself, send me another e-mail indicating what the expected audience numbers will be and I'll send you a special URL that you can distribute to the attendees for them to download a free ebook related to the theme of the meetup.

Software Architecture for Developers: Volume 1 - Technical leadership and the balance with agility

(image)

If you would like a copy of volume 1, try to organise a meetup related to the following topics: software architecture basics, the software architecture role, technical leadership, software architecture and agile, etc.

Software Architecture for Developers: Volume 2 - Visualise, document and explore your software architecture

(image)

And if you would like copies of volume 2, try to organise meetup related to the following topics: diagramming software architecture, the C4 model, documenting software architecture, exploring software architecture, etc.

Drop me a note if you have any questions.




Visualising and documenting software architecture cheat sheets

Thu, 27 Apr 2017 10:59:00 GMT

My cheat sheet summarising the C4 model has now been updated, and I've created another to summarise my thoughts on how to document software architecture. Click the images for the full-size (A3) PDF file.

(image) (image)

I hope you find them useful!




PlantUML and Structurizr

Thu, 08 Dec 2016 09:28:41 GMT

Despite the seemingly high unpopularity of UML these days, it continues to surprise me how many software teams tell me that they use PlantUML. If you've not seen it, PlantUML is basically a tool that allows you to create UML diagrams using text. While the use of text is very developer friendly, PlantUML isn't a modelling tool. By this I mean you're not creating a single consistent model of a software system and creating different UML diagrams (views) based upon that model. Instead, you're simply creating a single UML diagram at a time, which means that you need to take responsibility for the consistent naming of elements across diagrams.

Some teams I've worked with have solved this problem by writing small applications that generate the PlantUML diagram definitions based upon a model of their software systems, but these tend to be bespoke solutions that are never shared outside of the team that created them. Something I've done recently is create a PlantUML exporter for Structurizr. Using PlantUML in conjunction with the Structurizr open source library allows you to create a model of your software system and have the resulting PlantUML diagrams be consistent with that model. If you use the Structurizr's component finder, which uses reflection to identify components in your codebase, you can create component level UML diagrams automatically.

(image)

Even if you're not interested in using my Structurizr software as a service, I would encourage you to at least use the open source library to create a model of your software system, extracting components from your code where possible. Once you have a model, you can visualise that model in a number of different ways.




Introducing Structurizr Express

Mon, 06 Jun 2016 12:35:48 GMT

I rolled out a new feature to Structurizr at the weekend called Structurizr Express, which is basically a way to create software architecture diagrams using text. Although the core concept behind Structurizr is to create a software architecture model using code, there are times when you simply want a quick diagram, perhaps for a presentation, pre-sales proposal, etc. Structurizr Express will let you do just that - quickly create a single software architecture diagram using a textual definition. Much like tools such as PlantUML, yUML, WebSequenceDiagrams, etc.

(image)

Despite the name, this is all still based around the C4 model although it only targets one diagram at a time. The three types of diagrams currently supported are System Context, Container and Component diagrams. Structurizr Express is available to use now and the help page provides a description and examples of the syntax. I hope you find it useful.




Agile software architecture documentation

Tue, 31 May 2016 21:43:00 GMT

"We value working software over comprehensive documentation" is what the manifesto for agile software development says. I know it's now a cliche, but the typical misinterpretation of these few words is "don't write documentation". Of course, that's not actually what the manifesto says and "no documentation" certainly wasn't the intent. To be honest, I think many software teams never produced or liked producing any documentation anyway, and they're now simply using the manifesto as a way to justify their approach. What's done is done, and we must move on. One of the most common questions I get asked is how to produce "agile documentation", specifically with regards to documenting how a software system works. I've met many people who have tried the traditional "software architecture document" approach and struggled with it for a number of reasons, irrespective of whether the implementation was a Microsoft Word document or a wiki like Atlassian Confluence. My simple advice is to think of such documentation as being supplementary to the code, describing what you can't get from the code alone. Readers of my Software Architecture for Developers ebook will know that I propose something akin to a travel guidebook. Imagine you arrive in a new city. Without any maps or a sense of direction, you'll end up just walking up and down every street trying to find something you recognise or something of interest. You can certainly have conversations with the people who you meet, but that will get tiring really quickly. If I was a new joiner on an existing software development team, what I'd personally like is something that I can sit down and read over a coffee, perhaps for an hour or so, that will give me a really good starting point to jump into and start exploring the code. The software guidebook Although the content of this document will vary from team to team (after all, that's the whole point of being agile), I propose the following section headings as a starting point. Context Functional Overview Quality Attributes Constraints Principles Software Architecture Code Data Infrastructure Architecture Deployment Development Environment Operation and Support Decision Log The definitions of these sections are included in my ebook and they're now available to read for free on the Structurizr website (see the hyperlinks above). This is because the next big feature that I'm rolling out on Structurizr is the ability to add lightweight supplementary documentation into the existing software architecture model. The teams I work with seem to really like the guidebook approach, and some even restructure the content on their wiki to match the section headings above. Others don't have a wiki though, and are stuck using tools like Microsoft Word. There's nothing inherently wrong with using Microsoft Word, of course, in the same way that using Microsoft Visio to create software architecture diagrams is okay. But it's 2016 and we should be able to do better. Documentation in Structurizr The basic premise of the documentation support in Structurizr is to create one Markdown file per guidebook section and to link that with an appropriate element in the software architecture model, embedding software architecture diagrams where necessary. If you're interested to see what this looks like, I've pushed an initial release and there is some documentation for the techtribes.je and the Financial Risk System that I use in my workshops. The Java code and Markdown looks like this. Even if you're not using Structurizr, I hope that this blog post and publishing the definitions of the sections I typically include in my softw[...]