Subscribe: Sean's Obsessions
Added By: Feedage Forager Feedage Grade B rated
Language: English
add  app  application  book  chef  client  code  create  data  new  phone  rails  run  server  test  token  twilio  vault 
Rate this Feed
Rate this feedRate this feedRate this feedRate this feedRate this feed
Rate this feed 1 starRate this feed 2 starRate this feed 3 starRate this feed 4 starRate this feed 5 star

Comments (0)

Feed Details and Statistics Feed Statistics
Preview: Sean's Obsessions

Sean's Obsessions

Updated: 2016-12-21T01:43:55+00:00


Managing Secrets in Chef With Hashicorp Vault


The Problem It’s pretty common to need Chef to be able to store secrets such as database passwords or API keys. The easiest thing to do is to store them in a data bag, but that’s open for the world to see. Encrypted data bags are nice but key management is a gigantic pain. A better solution is Chef Vault, which encrypts the data bag’s secret once for each client (a client being a Chef node or administrative user) At the same time your organization likely has a need to keep secret data for applications, too. One could store these secrets in the same place as the Chef secrets but if you don’t like having Chef manage application configuration files then you’re out of luck. HashiCorp Vault is one solution here that we’ve used successfully. With HashiCorp Vault, each client (or groups of clients) has a token that gives them access to certain secrets, dictated by a policy. So you can say that a certain token can only read user accounts and give that to your application or Chef. But how do you keep that token a secret? I’ll also add that the management of HashiCorp Vault is nicer than that of Chef-Vault. That is, making changes to the secrets is a bit nicer because there’s a well defined API that directly manipulates the secrets, rather than having to use the Chef-Vault layer of getting the private key for the encrypted data bag and making changes to JSON. Furthermore this lets us store some of our secrets in the same place that applications are looking, which can be beneficial. In this example, I have a Chef recipe with a custom resource to create users in a proprietary application. I want to store the user information in HashiCorp vault because the management of the users will be easier for the operations team, and it will also allow other applications to access the same secrets. The basic premise here is that the data will go in HashiCorp Vault and the token to access the HashiCorp Vault will be stored in Chef’s Vault. The Code The first thing to do is set up your secrets in HashiCorp Vault. We’ll want to create a policy that only allows read access in to the part of the Vault that Chef will read from. Add this to myapp.hcl 1 2 3 path "secret/myapp/*" { policy = "read" } Create the policy: 1 2 [root@vault ~]# vault policy-write myapp myapp.hcl Policy 'myapp' written. Create a token that uses that policy. Note that the token must be renewable, as we’re going to have Chef renew it each time. Otherwise it’ll stop working after a month. 1 2 3 4 5 6 7 8 [root@vault ~]# vault token-create -policy=myapp -renewable=true Key Value --- ----- token ba85411e-ab76-0c0f-c0b8-e26ce294ae0d token_accessor token_duration 720h0m0s token_renewable true token_policies [myapp] That value beginning with ba85 is the token that Chef will use to talk to the Vault. With your root token you can add your first secret: 1 vault write secret/myapp/testuser password=abc123 path=/tmp At this point we have a user in the HashiCorp Vault and a token that will let Chef read it. Test for yourself with vault auth and vault read! Now it’s time to get Chef to store and read that key. Store the token in some JSON such as secret.json. 1 { "token": "ba85411e-ab76-0c0f-c0b8-e26ce294ae0d"} And create a secret that’s accessible to the servers and any people needed: 1 knife vault create myapp_credentials vault_token -A sean, -M client -J ./secret.json This creates a secret in a data bag called myapp_credentials in an item called vault_token. The secret itself is a piece of JSON with a key of token and a value of the token itself. The secret is only accessible by sean (me) and If you later want to add a new server or user to manage it, you need to run 1 knife vault update myapp_credentials vault_token -A Which will encrypt the data bag secret with something that only can decrypt. I won’t get into all the details of Chef Vault o[...]

Getting Started With Chef


I’ve been a proponent of configuration management with Chef for a while. It’s done amazing things for me and my workplace and I think everyone could benefit from it. But when I talk to people the question always comes up: “How do I get started? Things move slowly here.”. I’m going to share the plan that worked for me. YMMV. Note - While I talk about Chef, this also goes for Ansible, Salt, Puppet, cfengine, etc. The plan First, establish a beachhead. Get the Chef agent on all the servers you can, including your snowflakes. Then, start automating your “new box” workflow so that it’s as hands-off as possible and results in a fairly standardized build with Chef on it. Finally, commit to using those new boxes for everything you do. Once you have this done, you’ll immediately be able to prove the value of configuration management. You’ll be able to query Chef for things that normally took a while to get (who is running that old kernel version) and be able to automate many ad-hoc tasks (delete that account on all the servers). Over time you can improve to deploy your servers using cookbooks. Step 1: Chefify all the current infrastructure Install Chef Server on a separate box. The server is not necessary to get use out of Chef but it makes things easier. More importantly, once you finish this step you’ll immediately be able to store configuration management data and run queries on your infrastructure (once they run the Chef client). Next, create your first cookbook that will be a very sparse configuration that applies to all of your current infrastructure. When I did it I called it role-minimal and went down the role cookbook path. The TL;DR of that is that you create a role that only includes the role cookbook so that you get the benefits of versioning. What do you put in the minimal role? It can be nothing to start, if you want. Or maybe something that’s so benign that no-one could complain, like setting the banner or motd: 1 2 3 4 cookbook_file '/etc/motd' do source 'motd' mode '0644' end and then put your motd in files/default/motd. This will manage the file and ensure that all nodes have the same MOTD. Who can argue with that? You probably already have an item in your backlog to do that anyway. The other thing to add to this cookbook is to make the Chef client run on a schedule with the chef-client cookbook 1 2 3 4 5 6 7 8 9 10 default['chef_client']['init_style'] = 'none' default['chef_client']['cron'] = { minute: '*/30', hour: '*', path: nil, environment_variables: nil, log_file: '/dev/null', use_cron_d: false, mailto: nil } That can go in your attributes for the recipe to run it every half hour, or whatever you want. Don’t forget to include_recipe 'chef-client::cron' to have Chef manipulate your crontab to add the job. You may want to create environments if that’s the way you do things. After this, start bootstrapping your machines with knife bootstrap and a run list containing your new role. Maybe start with the non production servers. Don’t worry too much if people resist, you can leave those servers alone for now. Now you have Chef running in a low risk fashion. But what’s going to happen? Someone will eventually need something: We need to move to LDAP authentication We need to ensure that we’re never running an old version of openssl We need to know which servers are running which kernels And then you can put up your hand and say it’s easy, because you happen to have Chef agents on each server, so the easy way would be to leverage that. Except that server that they were concerned about earlier – did they want that one done by hand or should we use our new repeatable automated process? Great, I’ll just go bootstrap that node. Step 2: Fix your provisioning This one really depends on how you build new machines. The general idea is that you want to come up with a base machine configuration that everything you do from now on is buil[...]

LPI Certification Book Is Out


Ross Brunson and I spent a lot of time working on a study guide for the Linux+ and LPIC-1 certification, and I’m happy to say it’s finally out. Here’s an Amazon link.

I’m particularly proud of this one. It started out as an update to Ross’ book from 10 years ago, but in the end we rewrote most of it, expanded other sections, and added a ton of new content to cover the new objectives. Ross is a former employee of LPI so you know this book will cover the right material.

Canadian Equity Crowdfunding Rules


Be forewarned, I’m not a lawyer or a securities dealer. I’m just an interested person writing down my opinion.

Back in May the Canadian Security Administrators released the Start-up Crowdfunding Registration and Prospectus Exemptions (PDF). The core idea is that in certain situations a startup company can sell equity in their company without filing a detailed prospectus or registering as a dealer, often called crowdfunding, though here more properly called equity crowdfunding.

What’s the difference? In crowdfunding sites, such as Kickstarter, you give a project money in return for a prize or product. The project is assured a certain amount of sales to cover any capital costs, which should put them on a good footing for later sales. If the project is a success or a bust you don’t have any long term interest in the company – you’re basically pre-purchasing a product. In the equity crowdfunding scenario you’re buying a piece of the company much like if you bought it on the stock market. If the company does well then your investment may be worth a multiple of the purchase price. If the company does poorly then it’s worth nothing.

Normally these types of equity transactions are heavily regulated to prevent fraud and people from preying on novice investors. These new guidelines are an attempt to reduce the regulations while still preserving the investor protection. It should be noted that these only apply to British Columbia, Saskatchewan, Manitoba, Québec, New Brunswick and Nova Scotia. It is interesting to note that Ontario is developing their own rules.

While there are many conditions in these new equity crowdfunding guidelines the most important are:

  • The issuer (startup company) must write up an offering document containing basic information about the company and the money being raised. No financial statements are required, or even supposed to be attached to the offering document.
  • Each distribution (fundraising round) can be up to $250,000 and an issuer can only raise twice per calendar year.
  • A distribution declares a minimum amount it will raise, and it must raise that within 90 days or the money is returned to investors

Additionally the investors have some rights and limitations:

  • Can only contribute $1,500 per distribution
  • Has 48 hours to back out of a decision (either after the initial decision or after any updates have been made to the offering document)
  • May not be charged a fee for investing (i.e. the issuer must pay any fees)
  • Can not resell their shares unless
    • They are sold as part of another distribution
    • They are sold under a formal prospectus (e.g. what this system is trying to avoid)
    • The company becomes publicly listed and then only after a 4 month hold period expires

These distributions are expected to be sold through online portals that are either run by registered dealers or privately.

My take on all this is that it’s a good start. My main problem is the low limit on the personal contributions. If you raise $100k you need at least 67 people to buy in. I realize there must be a limitation to protect the downside but it seems it could have been a lot higher, maybe as much as $5,000. You now have 67 investors to manage, though you can make sure that the shares being issued to this class of shareholders has very few rights. If you go for institutional funding after a crowdfunded round then this may complicate things.

Thinking About Cyber Security


There have been a lot of high profile security breaches lately. If people like Sony can get hacked, what chance do you have? The people at Sony are humans. They get together at the water cooler and complain about the state of affairs and all the legacy applications they have to support. Even new companies like Dropbox are going to have dark corners in their applications. Money isn’t going to solve all these problems - Apple has billions of dollars and still gave up information about users. The bigger the company, the bigger the attack surface and the more the team has to defend. How do you prevent your company or product from being hacked given your resources are finite and you may not be able to change everything you want to? I’ve been thinking of how Agile methods such as rapid iteration and sprints could be applied to security. With that in mind, some high level principles: Solutions to problems should be ranked in terms of business value If a solution takes more than a week or two to implement, it should be broken down into individual phases with their own business value scores It’s not necessary to completely solve the problem as long as you’re better off than you were before. You can make it better the next iteration Instead of “how can we eliminate all risk?” the better question is “how can we make an attacker’s work more difficult?” Detection is just as important as prevention. Look at safes – they protect valuables against a determined adversary for a given period of time, it’s still up to you to make sure you can react in that timeframe The list above is trying to get away from the traditional security project where you spend lots of time producing documentation, shelve it, and then provide a half-assed solution to meet the deadline. Instead you break the solution into parts and try and continually produce business value. Time for a diagram: Even in the best case where you deliver full value, why not try to deliver parts of it sooner? Look at it this way – at any point in time you know the least amount about your problem as you ever will. It’s folly to think you can solve them all with some mammoth project. Pick something, fix it, move on. You have an end goal for sure, but the path may change as you progress. With that in mind, how do you figure out what to do? One organization technique I’ve found helpful is the attack tree. Here you define the goal of the attacker: Steal some money, take down the site, and so forth. Then you start coming up with some high level tasks the attacker would have to do in order to accomplish the goal. The leaf nodes are the more actionable things. For example, consider what it would take to deface a website: While not totally complete, this attack tree shows where the attack points are. Given that, some low effort and high value activities that could be done: Audit who has access to CDN, DNS, and registrar accounts Audit CMS accounts Some higher effort activities would then be: Code fix to enforce strong passwords Code fix to lock out accounts after a certain period Code fix to centralize the authentication to the corporate directory Investigate two factor or SAML login with hosting providers Network fix to ban IPs after a certain number of attempts Monitor failed login attempts and investigate manually Some of those options may be a lot of work. But if you first start with a simple password policy, build on that in the next iteration to lock out accounts, and finally tie in to another directory, you’re able to incrementally improve by making small fixes and learning as you go. What if a group like Anonymous threatens to attack your website on the weekend? Look at the attack tree, what kind of confusion can you throw at the attacker? Change the URL of the login page? Put up a static password on the web server to view the login screen itself? Security throu[...]

Test Driven Infrastructure


In software, test driven development happens when you write an automated test that proves what you are about to write is correct, you write the code to make the test pass, then you move on to the next thing. Even if you don’t follow that strict order (e.g. write your code, then write a test), the fact that there’s a durable test that you can run later to prove the system still works is very valuable. All the tests together give you a suite of tools to help prove that you have done the right thing and that regressions haven’t happened. What about the infrastructure world? We’ve always had some variant of “can you ping it now?”, or some high level Nagios tests. But there’s still some value to knowing that your test was good – if you make a change and then test, how can you be sure your test is good? If you ran the same test first you’d know it failed, then you could make your change. And then there’s the regression suite. A suite of tests that may be too expensive to run every 5 minutes through Nagios but are great to run to verify your change didn’t break anything. Enter the Bash Automated Test System - a Bash based test suite. It’s a thin wrapper around the commands that you’d normally run in a script but if you follow the conventions you get some easy to use helpers and an easy to interpret output. As an example, I needed to configure an nginx web server to perform a complicated series of redirects based on the user agent and link. I had a list of “if this then that” type instructions from the developer but had to translate them into a set of cURL commands. Once I had that it was simple to translate them into a BATS test that I could use to prove the system was working as requested and ideally share with my team so they could verify correctness if they made changes. share_link tests1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 #!/usr/bin/env bats @test "root" { run curl [[ $output =~ "doctype html" ]] } @test "mobile redirects to share" { run curl -H "User-Agent: this is an iphone" -i -k [[ $output =~ "302 Found" ]] [[ $output =~ "Location:" ]] } @test "mobile redirects to share and keeps query string" { run curl -H "User-Agent: this is an iphone" -i -k [[ $output =~ "302 Found" ]] [[ $output =~ "Location:" ]] } @test "desktop redirects to play" { run curl -H "User-Agent: dunno bob" -i -k [[ $output =~ "302 Found" ]] [[ $output =~ "Location:" ]] } @test "desktop redirects to play and keeps query string" { run curl -H "User-Agent: dunno bob" -i -k [[ $output =~ "302 Found" ]] [[ $output =~ "Location:" ]] } @test "bots redirect to main site" { run curl -H "User-Agent: facebookexternalhit" -i -k [[ $output =~ "302 Found" ]] [[ $output =~ "Location:" ]] } @test "bots redirect to main site and keeps query string" { run curl -H "User-Agent: facebookextern[...]

Using Google Universal Analytics With NationBuilder


We spent a lot of time trying to understand our visitors at Bowman for Winnipeg and part of that was using Google Analytics. The site was built with NationBuilder but they only support the async version of Analytics and it’s difficult to customize. In particular, we used the demographic and remarketing extensions and there was no easy way to alter the generated javascript to get it to work. Normally you’d just turn off your platform’s analytics plugins and do it yourself, but NationBuilder has a great feature that fires virtual page views when people fill out forms, and we wanted to use that for goal tracking. The solution was to turn off NationBuilder’s analytics and do it ourselves but write some hooks to translate any async calls into universal calls. Even with analytics turned off in our NationBuilder account, they’d fire the conversion events so this worked out well. In the beginning of our template: Header code1 2 3 4 5 6 7 8 9 10 11 12