Last Build Date: Wed, 04 Jan 2017 14:57:30 -0700Copyright: Copyright 2017
Wed, 04 Jan 2017 14:57:27 -0700Summary: Picos are a natural way to build microservices. This post presents the results of an experiment we ran to see how the new Pico Engine performs when placed under surge loading that simulates BYU's priority registration. Every semester over 30,000 BYU students register for two to six sections out of about 6,000 sections being offered. During the priority registration period this results in a surge as students try to register for the classes they need. One proposed solution for handling the surge is to create a microservice representing sections that can spawn copies of itself as needed. Thinking reactively, I concluded that we might want to just have an independent actor to represent each section. Actors are easy to create and the underlying system handles the scaling for you. Coincidentally, I happen to have an actor-style programming system. Picos are an actor-model programming system that supports reactive programming. As such they are great for building micorservices. Bruce Conrad and Matthew Wright are reimplementing the KRE pico platform using Node JS. We call the new system the "pico engine." I suggested to Bruce that this class registration problem would be a fun way to test the capabilities of the new pico engine. What follows is the results of Bruce's experiment. Experimental Setup We were able to get the add and drop requests for the first two days of priority registration for Fall 2016. This data included 6,675 students making 44,505 registration requests against 5,393 sections. The root actor in a pico system is called the Owner Pico. The Owner Pico is the ancestor of all other picos. For this experiment, Bruce created two child picos for the Owner Pico to represent the collection of sections and the collection of students. The Section Collection Pico is the parent of a pico for each section. The Student Collection Pico is the parent of a pico for each student. The resulting pico system looked like the following diagram from the Pico Engine's developer console: Overall Setup for Registration System (click to enlarge) Bruce chose not to create picos for each student for this experiment. He programmatically created picos for each section. The following figure shows the resulting pico graph: Picos for Every Section Taught in Fall 2016 (click to enlarge) Each section pico has a unique identity and knows things like it's name, section id, and seating capacity. The pico also keeps a roster of students who have registered for the course. The picos instantiated themselves with this data by calling a microservice with the data. Using this configuration, Bruce replayed the add/drop requests as events to the pico engine. Events were replayed by sending an section:add or section:drop event to the Section Collection Pico (which routed requests to the individual section picos). After the requests have been replayed, we can query individual section picos to ensure that the results are as expected. This figure shows the JSON result of querying the section pico for STDEV 150-3. Queries were done programmatically to verify all results. Querying a section pico shows it has the right state (click to enlarge) Results Bruce conducted two tests using a pico engine running on a Macbook Pro (2.6 GHz Intel Core i5). The first replay, done with per-event timing enabled, required just under an hour for the 44,504 requests. The average time per request was about 80 milliseconds (of which approximately 30 milliseconds was overhead relating to measuring the time). A second replay, done with the per-event timing disabled, processed the same 44,504 add/drops in 35 minutes and 19 seconds. The throughput was 21 registration events per second or 47.6 milliseconds per request. For reference, the legacy system, running on several large servers, sustains a peak rate of 80 requests per second, while averaging one request about every 3.9 seconds. These tests show that the pico engine can [...]
Fri, 30 Dec 2016 09:57:13 -0700Summary: GPG would make an excellent client for the Sovrin identity network and solve some of the problems that have prevented PGP from becoming a useful communication system. Lately, I’ve been thinking a lot about use cases for self-sovereign identity. This series of blog posts discusses how Sovrin can be used in different industries. In this article I discuss Sovrin and GPG. As I read I’m throwing in the towel on PGP, and I work in security from Filippo Valsorda, I couldn't help but think that it illustrates a real problem that Sovrin solves. Valdorda says: But the real issues, I realized, are more subtle. I never felt confident in the security of my long-term keys. The more time passed, the more I would feel uneasy about any specific key. Yubikeys would get exposed to hotel rooms. Offline keys would sit in a far away drawer or safe. Vulnerabilities would be announced. USB devices would get plugged in. A long-term key is as secure as the minimum common denominator of your security practices over its lifetime. It's the weak link. To see how Sovrin can help, let's talk about DIDs and DDOs. DIDs and DDOs A distributed ledger like Sovrin is a key-value store with entries that are temporally ordered and immutable. Decentralized Identifiers (DIDs) are intended to be one kind of key for a ledger. A DID is similar to a URN (universal resource name) and has a similar syntax. Here's an example DID: did:sov:21tDAKCERh95uGgKbJNHYp The ternary structure includes a schema identifier (did), a namespace identifier (sov in this case) and an identifier within that namespace (21tDAKCERh95uGgKbJNHYp) separated by colons. DIDs are meant to be permanently assigned and not reused. DIDs can be used to identify anything: person, organization, place, thing, or even concept. DIDs point to DDOs, or DID Descriptor Objects. A DDO is a JSON-LD-formatted data structure that links the DID to public keys, signatures, and service endpoints. We can use the signatures to validate that the DDO has not been tampered with, the service endpoint to discover more information about the DID, and the public key to secure communication with the entity identified by the DID. Sovrin is designed to support a different key pair for each DID. Consequently, a DID represents an identifier that can be specific to a particular relationship. Say I give 20 DIDs to friends I want to communicate with. Each associated DDO would contain a public key that I generated just for that relationship. I would sign these with a key that is well known and associated with social media and other well known online personas I maintain.1 The keys associated with these DIDs can be rotated as frequently as necessary since people never store my key, they only store the DID I give them for communicating with me. The ledger ensures that the most recent public key for a given DID can always be validated by checking the signature against the key associated with my well known DID. Of course, I've also stored DIDs for my friends and can check communications from them in the same way. These features, taken together, do away with the need for long-term keys and ease the burden of knowing the public key for someone you want to communicate with. So long as you know DID for the person, you can encrypt messages they can read. 2 A Proposal: GPG and Sovrin Which brings me to my proposal. Sometime in the first part of 2017, the Sovrin identity network will go live. The sandbox is available now. Many of the most advanced features of Sovrin will not be available in the MVP, but DIDs, DDOs, and public-private key pairs will be. Could GPG be modified to perform as a Sovrin Client? I believe the following would be required: Create DIDs with valid public keys on the Sovrin ledger3 Store and manage the private keys associated with those public keys Store DIDs and associate them with contacts on the user's computer Look up the DDO and public key[...]
Wed, 28 Dec 2016 17:57:52 -0700Summary: This article describes a method for using the Sovrin distributed identity ledger to lookup picos by name rather than location. This allows picos to be portable between hosting engines without loss of functionality. Lately, I’ve been thinking a lot about use cases for self-sovereign identity. This series of blog posts discusses how Sovrin can be used in different industries. In this article I discuss Sovrin and Picos. Anyone who reads this blog regularly knows that I've been working on a system for creating online agents using an event-driven, actor-model programming system called "persistent compute objects" or "picos" for short. Picos support a personal, decentralized model for the Internet of Things. Fuse, a connected-car platform, was designed to show how this model works. Picos support a hosting model that supports the same properties that the Internet itself has, namely decentralization, heterarchy, and interoperability. Central to these features is the idea of portability—a pico should be movable from one hosting platform (something we call a "pico engine") to another without loss of functionality. I believe that the future Internet of Things will require architectures where online "device shadows" (as Amazon calls them) are not only under the control of the owner, but hostable in a variety of places, not just on the manufacturer's servers. For example, I might buy a toaster that has a device shadow hosted by the Black and Decker and later decide to move that device shadow (along with its interaction history) to a hosting platform I control. Picos are a way of imagining postable device shadows that are under owner control. Names The most difficult problem standing in the way of easily moving picos between hosting engines is that picos receive messages using a URL that follows the format defined by the Sky Event Protocol. Because it's a URL, it's a location. Moving a pico from one engine to another requires updating all the places that refer to that pico, an impossible task. HTTP redirects could potentially be used, but they have scaling limitations I'd rather stay away from. Specifically, frequent moves might create long chains of redirects that have to be maintained since you can never be sure when every service with an interest in the pico had seen the redirect and updated their records. The more general solution is to have picos refer to each other by name and have some way of resolving a name to the current location of the pico. In the past this always led me to an uncomfortable reliance on some kind of centralized registry, but recent developments in distributed ledgers have made name resolution possible without reliance on a central registry. DIDs and DDOs A distributed ledger like Sovrin is a key-value store with entries that are temporally ordered and immutable. Decentralized Identifiers (DIDs) are intended to be one kind of key for a ledger. A DID is similar to a URN (universal resource name) and has a similar syntax. Here's an example DID: did:sov:21tDAKCERh95uGgKbJNHYp The ternary structure includes a schema identifier (did), a namespace identifier (sov in this case) and an identifier within that namespace (21tDAKCERh95uGgKbJNHYp) separated by colons. DIDs are meant to be permanently assigned and not reused. DIDs can be used to identify anything: person, organization, place, thing, or even concept. DIDs point to DDOs, or DID Descriptor Objects. A DDO is a JSON-LD-formatted data structure that links the DID to public keys, signatures, and service endpoints. We can use the signatures to validate that the DDO has not been tampered with, the service endpoint to discover more information about the DID, and the public key to secure communication with the entity identified by the DID. Using Sovrin with Picos DIDs on the Sovrin ledger provide a convenient way to name picos with an identifier that can be resolved to the current location of the pico without relying on a cen[...]
Wed, 07 Dec 2016 19:51:41 -0700Summary: Sovrin's verifiable claims provide the means of creating a virtual university with little or no traditional integration between the various players. Lately, I’ve been thinking a lot about use cases for self-sovereign identity. This series of blog posts discusses how Sovrin can be used in different industries. In this article I discuss Sovrin and education. Last spring I wrote about how deconstructing the student information system at BYU is allowing us to create a virtual university. The idea is to create a decentralized system of student profiles that contain both profile data as well as learning records. These learning records can attest to any kind of learning activity from the student having read a few paragraphs to the completion of her degree. By making the student the point of integration, we avoid having to do heavy, expensive point-to-point integrations between all the student information systems participating in the educational initiative. This diagram shows a number of players in the virtual university (VU) system (labeled CES Initiative). They have no direct connections to each other, but present their own APIs, as does the student profile. As students interact with the various institutions, they present data in their profile (using an API). That's the only point of integration for these institutions, including VU itself. One of the design requirements for the student profile is that it be hostable where ever the student desires. We also put the student in control of authorizations to use data. These features mitigate regulatory requirements around privacy of student data. With millions of potential student spread among 188 different countries, this saves significant cost and headache trying to meet various regulatory hurdles. When the student is an active participant in sharing their data, the hurdles can be more easily overcome. The Role of Self-Sovereign Identity Self-sovereign identity can simplify many of the most difficult challenges of building the student profile. One of the key features of Sovrin is support for verifiable claims shared by their subject. Nearly everything in the student profile looks like a verifiable claim. Sovrin provides the infrastructure for issuing claims and using them in disclosures. Sovrin enables the receiver of the disclosure to verifying the claims in it. Verifiable claims also significantly ease the integration burden between the various systems involved in the virtual university. Say Alice wants to be a book keeper, but the book keeping microcourse has requirements for English language proficiency and secondary school completion, that she lacks. There could be four institutions involved in this scenario: Virtual University (VU), the English language certifier (ELC), the secondary education provider (SEP), and the college offering book keeping (BK). These last three institutions all have agreements with VU, but no technical integration with VU or each other. Alice has be admitted to VU and paid for the book keeping microcourse along with the require pre-requisites.1 She goes to ELC. Alice can prove she's a VU student and has paid for the English language certification by disclosing that information from the verifiable claims that VU wrote to her student profile. ELC can validate that the disclosure is true without talking VU directly. Alice registers for a course and, after she completes it, ELC writes a claim that she passed to her student profile. Alice, guided by the Student Profile, heads to SEP and discloses that she's a VU student. SEP validates her student status and that she's paid for the SEP program. They ask her to prove that she has the required level of english competancy and she provides a disclosure based on the claim that ELC wrote. She completes the secondary education program and SEP writes a claim stating such.2 The same thing happens for Alice's book keeping course at BK. One notable difference is that the microcourse th[...]
Thu, 17 Nov 2016 16:10:10 -0700Summary: The following are my live tweets from Defrag at the Omni Interlocken in Denver, November 16-17, 2016. The following are my live tweets from Defrag at the Omni Interlocken in Denver, November 16-17, 2016. Eric Norlin: Defrag is 10 years old. Happy birthday. Announcing that this is the last time Defrag exists independently. Combining with Glue. #sad Tim Wagner: Going server less with AWS lambda Does serverless mean no servers? No, but there are no software servers (web server framework, etc.) Serverless goes beyond VMs and containers so that functions are the unit of abstraction Serverless enforces good design: small individual units of code, persistence separated from compute, one way to do things Triggers by events or calls from APIs. Easy to do real-time processing, event processing Bring your own code, Lambda is the web server, uses simple resource model, processes are stateless Economic proposition for lambda is "never pay for idle" Lambda economic model makes building a service simple by removing need to understand servers, availability zones, scaling Would you rather carry a giant beachball of 700 marbles without a bag. Both suck. Monoliths and microservices... Lambda provides the bag for the marbles.pen spec and tools to package and deploy entire serverless model #comingsoon Serverless means bring your own frameworks. No need to learn a new framework Lambda is a universal webhook receiver platform. Lambda reduces the need for understanding complex distributed computing concepts Future is a hybrid of containers and serverless to bridge these two worlds Prediction: we go from abstracting compute infrastructure to abstracting computational systems Lorinda Brandon: Marketing person at a tech conference gets "you have stickers, right?" Marketing smells funny: they always have to good news, they're perky. Makes you think they're hiding something Marketing thinks developers are mean and scary. developers are great conversationalists. Developer marketing has to take the bad with the good. Get feedback Tips for developer marketing: Start with GOOD; respond with truth; actively listen be flexible; get out of the way API == assume positive intent Duncan Johnston-Watt: Hyperledger is a project of the Linux foundation; lots of fancy sponsors A world of many chains. There will not be a chain of all chains. There will be many public chains and millions of private chains Blockchain allows everyone to agree on who owns what (for example) rather than having central registry for the asset Hyperledger fabric is permissioned blockchain network written in Go, chaincode in Go or Java Cloudsoft's interest in Hyperledger is deploying managing the consensus network and deploying applications on it. For example: Sotheby's want to run an auction. Cloudsoft could manage the deployment and management of Sotheby's auction code Mike Zaccardo: Now Mike Zaccardo is demoing the Sotheby's auction demo Hyperledger fabric has a notion of containers that allow multiple blockchain apps to run on the Hyperledger fabric demo shows transferring lots from one client to another, ensuring ownership Duncan Johnston-Watt Hyperledger is too big to be owned by a single entity. We should own it in common. See http://www.cloudsoft.io/gethlf for more info Tags: defrag notes conferences [...]
Mon, 14 Nov 2016 14:11:09 -0700Summary: Sovrin can improve healthcare and make it less costly by providing an identity system that combines a secure means of exchanging verifiable claims and patient consent that is a structural component of the system. Lately, I've been thinking a lot about use cases for self-sovereign identity. This is the second in a series of blog posts that discuss how Sovrin can be used in different industries. In this article I discuss Sovrin and healthcare. Healthcare is broken and many look to information technology to help solve the problem. But that's not working out as well as we'd like. The root of the problem, like many, is identity. Self-sovereign identity is the solution. Consent and Privacy One of the fundamental tenants of healthcare is patient consent. Patients have the right to determine their care and treatment. And they have a right to privacy in their healthcare. But consent and privacy are not the strong suits of today's Web architectures. Consequently, they're a poor match for building online patient services and it shows. Take two examples, the patient portal and the health information exchange. Patient Portals If you've been to the doctor lately, you've probably been directed to their "patient portal." What a nightmare. The security is too good, forcing you through their generally awful user experience just to perform tasks like reading short, low-value messages. As Bruce Fryer notes, these portals don't link to each other and they use name and date of birth to probabilistically correlate patients across systems. Health Information Exchanges A health information exchange (HIE) is another new piece of healthcare software that "allows doctors, nurses, pharmacists, other health care providers and patients to appropriately access and securely share a patient's vital medical information electronically—improving the speed, quality, safety and cost of patient care." These are supposed to help with interoperability between patient portals, but without clear identifiers for each patient, it's hit and miss. What's more, as Dr. Adrian Gropper points out, current HIEs store massive amounts of patient data with little accountability and don't properly allow patients to control access to data or grant consent for specific uses. Adrian has rethought how a HIE could work to put the patient in the consent flow and has a wonderful concept he calls HIE of One. He's got numerous demos to illustrate different flows. Adrian's HIE of One solves the problem of exchanging information between healthcare professionals while allowing for patient consent and control by putting the patient in the flow. This is a huge step forward and shows that decentralized solutions can better solve consent problems than centralized solutions. Self-sovereign identity systems plug right into the HIE of One to provide patient credentials, but that's just table stakes. Using a system like Sovrin, a patient-centric HIE would provide verifiable claims about patient data including test results, clinical findings, past treatment, and drug history. Insurance companies and other payers could also use claims to exchange critical information in a trustworthy, secure manner. Sovrin allows for disclosure and records consent. These claims would be anchored in the Sovrin ledger to ensure they can be validated. The Role of Self-Sovereign Identity The healthcare industry has reopened the universal patient identifier (UPI) debate as a potential solution to the problem of correlating patients across portals and exchanges, but there are several problems with this. First, an UPI would be subject to all the same kinds of problems that the social security number (SSN) has been. Once there's a single identifier anyone can correlate patient activity wherever it shows up. This puts privacy at the mercy of agreements rather than providing structural mechanisms to support privacy. Second, an UPI doesn't solve the inte[...]
Thu, 03 Nov 2016 18:13:25 -0600Summary: This use case discusses authentication in a self-sovereign identity system called Sovrin. Sovrin simplifies authentication, reducing friction while providing a system that businesses can trust without building or maintaining it. Lately, I've been thinking a lot about use cases for self-sovereign identity. This is the first in a series of blog posts that discusses how Sovrin can be used for various purposes. We'll begin with authentication. Authentication has several facets. I’m going to discuss two of them below: enrollment and login. In this use case, a person named Alice will be enrolling and logging into a service provided by Bob’s Bait Shop. Enrollment How does Alice get a self-sovereign identity? If you read my How Sovrin Works article, you should be aware that there’s no such thing as “an identity” in Sovrin. Instead, Sovrin creates a unique identifier for each relationship that Alice has. Alice controls her identifiers and can correlate them, but no one else can without her permission. So, enrollment consists of Alice generating a key pair for a new relationship and the Bob’s Bait Shop associating that with an account. There are different ways this could work, but let’s explore just one that involves a Sovrin app called a connector. The connector runs on the person’s device and helps them manage their keys. In some ways, it would feel like a password management app such as 1Password. The connector is secured by the operating-system security services on Alice's device as well as app-specific mechanisms. When Alice wants to connect with Bob’s Bait Shop via a Web site or mobile app, she uses the connector to generate a new Sovrin public-private key pair for that relationship. The connector securely stores the private key, then uses the public key to generate and register a new identifier on Sovrin. Alice gives the identifier to Bob’s Bait Shop, who creates an account and associates the internal account identifier with the Sovrin identifier. This association could be memorialized as a claim. If desired, Bob’s Bait Shop could generate a challenge that Alice signs with her connector to prove that she gave them a real Sovrin identifier. Bob’s Bait Shop might also ask Alice to provide information for creating the account. The connector would automatically determine which claims (either verifiable or self-asserted) can be used to satisfy the request, obtain permission from Alice, and then transmit those claims to Bob’s Bait Shop. Sovrin can partially disclose the information in the claims to protect Alice’s privacy. As part of requesting information from Alice, Bob’s Bait Shop could send a consent receipt that records promises made to Alice about how her disclosed information will be used. The connector stores a hash of the consent receipt on the Sovrin ledger to notarize it. Note that Alice didn’t create a password or fill out forms. And she received a receipt documenting her consent. The connector remembers all of the services with which she’s enrolled and gives her a single place where she can see everyone with whom she has a relationship1. Bob’s Bait Shop doesn’t store or manage passwords. They didn’t have to validate email addresses or phone numbers (provided they received claims they trust). And yet, they have a secure method of authenticating Alice and account information they can trust. Login User authentication is the most obvious use case for any identity system. Based on the discussion of enrollment above, assume Alice and Bob’s Bait Shop have already exchanged an identifier and that Alice’s connector is storing the corresponding private key as well as an identifier for Bob’s Bait Shop that can be used as a public key. Authentication is done using any of the standard public-private key authentication schemes. One method consists of Bob’s Bait Shop pr[...]
Wed, 02 Nov 2016 08:30:07 -0600
Summary: Sovereignty is much more than control. Sovereignty is about relationships and balance of power.(image)
There's a lot of confusion about self-sovereign identity.
Many people hear sovereign and think "sovereign means the individual has complete control." Not really. As Scott David pointed out, "declaring yourself king of a deserted island isn't very useful."
Sovereignty is about relationships and boundaries. When we say a nation is sovereign, we mean that it can act as a peer to other sovereign states, not that it can do whatever it wants. Sovereignty defines a boundary, within which the sovereign has complete control and outside of which the sovereign relates to others within established rules and norms.
Self-sovereign identity (SSI) describes the same situation. A self-sovereign identity system (SIS) defines the things over which an entity (person or organization) has complete control along with the rules of engagement for its relations with other entities in the SIS system.
For example, an SIS might give entities complete control over what claims they share in response to queries about them. But sovereignty doesn't mean that a party relying on a claim has to accept it. The relying party's sovereignty allows it to determine what claims satisfy its demands and which don't.
But sovereignty means that all powers are reciprocal. Continuing the example, anyone can reject the claims others choose to present to them. The key to sovereignty is that all entities are peers. I have the same rights you do. The beauty of sovereignty isn't complete and total control, but rather balance of power that leads to negotiations about the nature of the relationships between various entities in the system.
Thu, 20 Oct 2016 14:32:45 -0600Summary: Verifiable, owner-provided attributes are the engine that will drive wide-spread adoption of self-sovereign identity systems. This article explains how this models the way credentials work in the physical world and describes benefits of owner-provided attributes. In the physical world, people carry credentials to prove to others that they have certain attributes or hold certain privileges. Online, this is not possible. For example, a driver's license contains attributes like name, address, and date of birth that have been validated by the Driver's License Division. The driver’s license is widely viewed as trustworthy. Consequently, people use driver's licenses for purposes other than driving. For example, a school or pharmacy can easily verify that a license belongs to the person presenting it, and confirm the validity of the license without ever contacting the Driver's License Division directly. In other words, in the physical world, people hold and are the conveyors of their own trustworthy attributes (called “claims” by identity experts). These attributes can be used when needed and without prior agreement. Online, such interactions are only possible through pre-arranged integrations between the APIs of specific computer systems. Identity systems in use today include federation for business-to-business credential sharing, and social login for consumer authentication1. Neither of these offers a foundation upon which third-party claim issuers can easily build services that allow for the kind of ad hoc attribute sharing that happens in the physical world. Consequently, entities who want to rely on attributes from many parties have to perform integrations with all of them. This is slow, complex, and costly, so it typically happens only for high-value applications. Decentralized identity systems, like Sovrin, have built-in support for third-party claims that function in the same way physical credentials work: they’re presented directly by the identity owner. The identity owner can use a claim from one party to disclose attributes to another party without those parties even having a relationship, much less a technical integration2. Claims can be used in ad hoc situations, just as they can in the physical world, allowing individuals to function as integration points. When you can instantly trust what someone says about themselves, workflows and integrations are dramatically simplified. There are other benefits to owner-provided attribute sharing. First, when owners share attributes, the receiver automatically gains consent to use the attributes for the requested purpose. This significantly reduces liability. Second, when the owner is part of the process, they can check the accuracy of the attributes as they’re being shared, resulting in better data. Owner-provided attributes are a powerful driver that will push decentralized identity systems well beyond the current uses of federation and social login. Businesses can reduce or even eliminate API integrations and manual verification processes, and instead trust what's presented to them by customers, because the claims can be verified. Customers become the source of what's true about them. Businesses will find great value in this, driving adoption by individuals as customers are brought into decentralized identity systems through day-to-day business activities. See Self-Sovereign Identity and the Legitimacy of Permissioned Ledgers for more on the difference between self-sovereign identity systems and social login. See How Sovrin Works for details on how claims can be received, stored, and shared as verifiable disclosures. Tags: distributed+ledger identity sovereign-source claims sovrin [...]
Fri, 07 Oct 2016 14:55:41 -0600
Summary: An animation showing how Sovrin works.
If you're too busy to read How Sovrin Works, here's a short animation that summarizes the post using my graphics. Thanks to Andy Tobin for putting it together.(image)