Last Build Date: Mon, 24 Oct 2016 06:08:06 PDTCopyright: Copyright 2011
Mon, 24 Oct 2016 09:00:00 PDT
I am excited that Adrian Cockcroft will be joining AWS as VP of Cloud Architecture. Adrian has played a crucial role in developing the cloud ecosystem as Cloud Architect at Netflix and later as a Technology Fellow at Battery Ventures. Prior to this, he held positions as Distinguished Engineer at eBay and Sun Microsystems. One theme that has been consistent throughout his career is that Adrian has a gift for seeing the bigger engineering picture.
At Netflix, Adrian played a key role in the company's much-discussed migration to a "cloud native" architecture, and the open sourcing of the widely used (and award-winning) NetflixOSS platform. AWS customers around the world are building more scalable, reliable, efficient and well-performing systems thanks to Adrian and the Netflix OSS effort.
Combine Adrian's big thinking with his excellent educational skills, and you understand why Adrian deserves the respect he receives around the world for helping others be successful on AWS. I'd like to share a few Adrian's own words about his decision to join us....
"After working closely with many folks at AWS over the last seven years, I am thrilled to be joining the clear leader in cloud computing.The state of the art in infrastructure, software packages, and services is nowadays a combination of AWS and open source tools. -- and they are available to everyone. This democratization of access to technology levels the playing field, and means anyone can learn and compete to be the best there is."
I am excited about welcoming Adrian to the AWS team where he will work closely with AWS executives and product groups and consult with customers on their cloud architectures -- from start-ups that were born in the cloud to large web-scale companies and enterprises that have an “all-in” migration strategy. Adrian will also spend time engaging with developers in the Amazon-sponsored and supported open source communities. I am looking really looking forward to working with Adrian again and seeing the positive impact he will have on AWS customers around the world.
Mon, 17 Oct 2016 09:00:00 PDT
Today I am very happy to announce the opening of the new US East (Ohio) Region. The Ohio Region is the fifth AWS region in the US. It brings the worldwide total of AWS Availability Zones (AZs) to 38, and the number of regions globally to 14. The pace of expansion at AWS is accelerating, and Ohio is our third region launch this year. In the remainder of 2016 and in 2017, we will launch another four AWS regions in Canada, China, the United Kingdom, and France, adding another nine AZs to our global infrastructure footprint.
We strive to place customer feedback first in our considerations for where to open new regions. The Ohio Region is no different. Now customers who have been requesting a second US East region have more infrastructure options for running workloads, storing files, running analytics, and managing databases. The Ohio Region launches with three AZs so that customers can create high-availability environments and architect for fault tolerance and scalability. As with all AWS AZs, the AZs in Ohio each have redundant power, networking, and connectivity, which are designed to be resilient to issues in another AZ.
We are also glad to offer low transfer rates between both US East Regions. Data transfer between the Ohio Region and the Northern Virginia Region is priced the same as data transfer between AZs within either of these regions. We hope this will be helpful for customers who want to implement backup or disaster recovery architectures and need to transfer large amounts of data between these regions. It will also be useful for developers who simply want to use services in both regions and move resources back and forth between them. The Ohio Region also has a broad set of services comparable to our Northern Virginia Region, including Amazon Elastic Compute Cloud (Amazon EC2), Amazon Simple Storage Service (Amazon S3), Amazon Relational Database Service (Amazon RDS), and AWS Marketplace. Check out the Regional Products and Services page for the full list.
We’ll continue to add new infrastructure to grow our footprint and make AWS as useful as possible for all of our customers around the world. You can learn more about our growing global infrastructure footprint at https://aws.amazon.com/about-aws/global-infrastructure/.
Wed, 12 Oct 2016 22:00:00 PDTFast Data is an emerging industry term for information that is arriving at high volume and incredible rates, faster than traditional databases can manage. Three years ago, as part of our AWS Fast Data journey we introduced Amazon ElastiCache for Redis, a fully managed in-memory data store that operates at sub-millisecond latency. Since then we’ve introduced Amazon Kinesis for real-time streaming data, AWS Lambda for serverless processing, Apache Spark analytics on EMR, and Amazon QuickSight for high performance Business Intelligence. While caching continues to be a dominant use of ElastiCache for Redis, we see customers increasingly use it as an in-memory NoSQL database. Developers love the blazing fast performance and in-memory capabilities provided by Redis, making it among the most popular NoSQL key-value stores. However, until now ElastiCache for Redis customers could only run single-shard Redis. This limited the workload size and write throughput to that of a single VM, or required application level sharding. Today, as a next step in our Fast Data journey, we have extended the ElastiCache for Redis service to support “Redis Cluster,” the sharding capability of Redis. Customers can now scale a single deployment to include up to 15 shards, making each Redis-compatible data store up to 3.5 terabytes in size, that operate on microsecond time scales. We also do this at very high rates: up to 4.5 million writes per second and 20 million reads per second. Each shard can include up to five read replicas to ensure high availability so that both planned and unforeseen outages of the infrastructure do not cause application outages. Building upon Redis There are some great examples and use cases for Redis, which you can see at companies like Hudl, which offers mobile and desktop video analytics solutions to sports teams and athletes. Hudl is using ElastiCache for Redis to provide millions of coaches and sports analysts with near real-time data feeds that they need to help drive their teams to victory. Another example is Trimble, a global leader in location services who is using ElastiCache for Redis as their primary database for workforce location, helping customers like DirecTV get the right technician to the right location as quickly and inexpensively as possible, enabling both reduced costs and increased satisfaction for their own subscribers. Increasingly, ElastiCache for Redis has become a mission critical in-memory database for our customers whose availability, durability, performance and scale matter to their business. We have therefore been enhancing the Redis engine running on ElastiCache for the last few years using our own expertise in making enterprise infrastructure scalable and reliable. Amazon’s enhancements address many day-to-day challenges with running Redis. By utilizing techniques such as granular memory management, dynamic I/O throttling and fine grained replica synchronization, ElastiCache for Redis delivers a more robust Redis experience. It enables customers to run their Redis nodes at higher memory utilization without risking swap usage during events such as snapshotting and replica synchronization. It also offers improved synchronization of replicas under load. In addition, ElastiCache for Redis provides smoother Redis failovers by combining our Multi-AZ automated failover with streamlined synchronization of read replicas. Replicas now recover faster as they no longer need to flush their data to do a full resynchronization with the primary. All these capabilities are available to customers at no additional charge, and maintain open-source Redis compatibility. With this launch, we augmented the client-based failover logic of Redis 3.2 with ElastiCache for Redis Multi-AZ. If a customer is running a self-managed Redis environment on EC2 instead of using ElastiCache for Redis and the primary node fails, the cluster relies on a majority of primaries to determine and execute a failover. If such a majority doesn’t exist, the cluster will [...]
Thu, 29 Sep 2016 10:00:00 PDTIn the past voice interfaces were seen as gimmicks, or a nuisance for driving “hands-free.” The Amazon Echo and Alexa have completely changed that perception. Voice is now seen as potentially the most important interface to interact with the digitally connected world. From home automation to commerce, from news organizations to government agencies, from financial services to healthcare, everyone is working on the best way is to interact with their services if voice is the interface. Especially for the exciting case where voice is the only interface. Voice makes access to digital services far more inclusive than traditional screen-based interaction, for example, an aging population may be much more comfortable interacting with voice-based systems than through tablets or keyboards. Alexa has propelled the conversational interface forward given how natural the interactions are with Alexa-enabled devices. However, it is still Day One, and a lot of innovation is underway in this world. Given the tremendous impact of voice on how we interact with the digital world, it influences how we will build products and services that can support conversations in ways that we have never done before. As such there is also a strong need for fundamental research on these interactions, best described as “Conversational Artificial Intelligence.” Today, we are pleased to announce the Alexa Prize, a $2.5 million university competition to accelerate advancements in conversational AI. With this challenge, we aim to advance several areas of conversational AI including knowledge acquisition, natural language understanding, natural language generation, context modeling, commonsense reasoning and dialog planning. The goal is that through the innovative work of students, Alexa users will experience novel, engaging conversational experiences. Teams of university students around the world are invited to participate in a conversational AI challenge (see contest rules for details). The challenge is to create a socialbot, an Alexa skill that converses with users on popular topics. Social conversation can occur naturally on any topic, and teams will need to create an engaging experience while maintaining relevance and coherence throughout the interaction. For the grand challenge we ask teams to invent a socialbot smart enough to engage in a fun, high quality conversation on popular societal topics for 20 minutes. As part of the research and judging process, millions of Alexa customers will have the opportunity to converse with the socialbots on popular topics by saying, “Alexa, let’s chat about (a topic, for example, baseball playoffs, celebrity gossip, scientific breakthroughs, etc.).” Following the conversation, Alexa users will give feedback on the experience to provide valuable input to the students for improving their socialbots. The feedback from Alexa users will also be used to help select the best socialbots to advance to the final, live judging phase. The team with the highest-performing socialbot will win a $500,000 prize. Additionally, a prize of $1 million will be awarded to the winning team’s university if their socialbot achieves the grand challenge of conversing coherently and engagingly with humans for 20 minutes. Teams of university students can submit applications now and the contest will conclude at AWS re:Invent in November 2017, where the winners will be announced. Up to ten teams will be sponsored by Amazon and receive a $100,000 stipend, Alexa-enabled devices, free AWS services and support from the Alexa team. Participating teams will receive special access to new Alexa Skills Kit (ASK) APIs to build their skills. Registration opened today and teams have until October 28, 2016 to submit their applications. The competition will officially start on November 14, 2016 and run until November 2017, concluding with an award ceremony to be held at AWS re:Invent in Las Vegas, NV. For more information, check out the Alexa Prize page[...]
Thu, 29 Sep 2016 00:00:00 PDTToday, I am very excited to announce our plans to open a new AWS Region in France! Based in the Paris area, the region will provide even lower latency and will allow users who want to store their content in datacenters in France to easily do so. The new region in France will be ready for customers to use in 2017. Over the past 10 years, we have seen tremendous growth at AWS. As a result, we have opened 35 Availability Zones (AZs), across 13 AWS Regions worldwide. We have announced several additional regions in Canada, China, Ohio, and the United Kingdom – all expected in the coming months. We don’t plan to slow down or stop there. We are actively working to open new regions in the locations our customers need them most. French organizations were amongst the first to use AWS when we launched in 2006. Since we opened the first AWS EU Region in Ireland in November 2007, we have seen an acceleration of companies adopting the AWS Cloud. To support our customers’ growth, their digital transformation, and to speed up their innovation and lower the cost of running their IT, we continue to build out additional European infrastructure. Our CDN and DNS network now has 18 points of presence across Europe, we have added a third AZ in Ireland, a second infrastructure region in Frankfurt and a third region in the UK (due in coming months). After the launch of the French region there will be 10 Availability Zones in Europe. We have also expanded our presence in France over the last ten years. We have launched three points of presence, with two in Paris and one in Marseille, and also opened offices in the country, employing account managers, solutions architects, trainers, Business Development and Professional Services teams, as well as other job functions. Our teams are helping companies of all sizes, operating in various industries, such as finance, business, media, and many others, move to the cloud. As a result, more than 80 percent of companies listed on the CAC 40, the French stock market index, are now using AWS Cloud technology to speed their time-to-market, lower their costs, and support their businesses globally. Within the thousands of businesses using AWS in France, we count enterprises such as Schneider Electric, Lafarge and Dassault Systemes as customers as well as CAC40, multinational bank, Societe Generale Group. When we first talked to Societe Generale Group about opening the AWS region, Carlos Goncalves, Head of Global Technology Services, said, "We are delighted to learn that Amazon Web Services will open a region in France. Using the AWS Cloud, and the extended services offered by the platform, is an opportunity for us to accelerate our transformation and focus on how we can better serve our clients.” Another CAC40 company using the cloud to support its digital transformation is Veolia Water France, a subsidiary of Veolia, specialized in the distribution and the treatment of water. In the past we have had Benito Diz, CIO Veolia Water France speak at our events where he has talked about how they have been able to achieve important cost reductions while improving security and agility by moving to AWS. He has said, “By moving a large part of our IT system from our old IBM mainframe to AWS, we have adopted a cloud first strategy, boosting our power of innovation. By launching a new platform to analyze the Terabytes of data collected by the sensors located in our thousands of water meter or water vats we are creating an Internet of Things (IoT) system that helps us to reduce the maintenance intervention time, anticipate the refills and have in real time the information on the key indicators (temperature, water purity, pH level ...). We couldn’t have launched this industrial IoT project without the AWS flexibility.” width="640" height="360" src="https://www.youtube.com/embed/v9s6ClXJNxs?rel=0&showinfo=0" frameborder="0" allowfullscreen> In other sectors, government organizations, as w[...]
Tue, 27 Sep 2016 23:00:00 PDT
I am very grateful that I have had the opportunity to meet with President Shimon Peres several times. Especially the first time, which was a 1:1 in his presidential residence, was an unforgettable experience. After I explained in 5 minutes the power of cloud for unlocking digital business building for everyone, he went on a lecture of half an hour how bringing economic prosperity to the region was crucial to achieving a long lasting peace. "A hungry neighbor is an angry neighbor".
He strongly believed peace in the Middle East was attainable, and I have no reason to doubt him. If it will happen one day it will be because of believers like him.
RIP Mr. President.
Mon, 27 Jun 2016 05:00:00 PDTIntroducing New Features That Make It Easier for Customers to Discover and Use Your Alexa Skills Alexa, Amazon’s cloud-based voice service, powers voice experiences on millions of devices, including Amazon Echo and Echo Dot, Amazon Tap, Amazon Fire TV devices, and devices like Triby that use the Alexa Voice Service. One year ago, Amazon opened up Alexa to developers, enabling you to build Alexa skills with the Alexa Skills Kit and integrate Alexa into your own products with the Alexa Voice Service. Today, tens of thousands of developers are building skills for Alexa, and there are over 1,400 skills for Alexa – including Lyft and Honeywell, which were added today. A New Experience for Discovering Skills Today, we announced new ways for customers to discover and use the Alexa skills that developers have built, including a new voice-enablement feature and a completely redesigned Alexa app. Customers can now quickly search, discover and use skills. Starting today, customers can browse Alexa skills by categories such as “Smart Home” and “Lifestyle” in the Alexa app, apply additional search filters, and access their previously enabled skills via the “Your Skills” section. Also available today, Alexa customers can use their voice to enable your skills: simply say “Alexa, enable NBC News” or “Alexa, enable 7 Minute Workout” and access them instantly. Customers can also find your skills with Amazon’s Skill Finder. To use Skill Finder, simply enable it via voice or in the Alexa app and say "Alexa, ask Skill Finder for the top skills." One-Year Anniversary: ASK, AVS and The Alexa Fund In addition to the new Alexa skill features, June 25th marked the one-year anniversary of our developer services. Last June, we introduced the Alexa Skills Kit (ASK), the Alexa Voice Service (AVS), and the Alexa Fund to help enable anyone to build the experience they wanted for Alexa. Some fun facts about the Alexa Skills Kit, Alexa Voice Service, and Alexa Fund include: There are now over 1,400 Alexa skills and the catalog has grown by 50% in just over one month Customers have made over 3 million requests using the top 10 most popular Alexa skills Since January 2016, selection of Alexa smart home API skills has grown by more than 5x There are now over 10,000 registered developers using the Alexa Voice Service to integrate Alexa into their products There are tens of thousands of developers currently working on Alexa projects The Alexa Fund has invested in 16 startups, with a focus on smart home and wearable products to date. Over the next year, The Alexa Fund will be expanding investments into startups that focus on robotics, developer tools, healthcare, accessibility and more Some of the most popular Alexa skills are Jeopardy!, Daily Affirmation, Magic 8 Ball, Fitbit, and The Bartender Build a Skill Today - Special Offers Our skill templates and step-by-step guides are a valuable way to quickly learn the end-to-end process for building and publishing an Alexa skill. You can get started quickly using the flash cards skill template, fact skill template, trivia skill template, or how to skill template. Plus, if you publish a skill, you’ll receive an Alexa dev t-shirt. Quantities are limited. See Terms and Conditions. Additional Resources For more information on getting started with devloping for Alexa, check out the following resources: Alexa Developer Platform Alexa Skills Kit (ASK) Alexa Voice Service (AVS) The Alexa Fund ASK Developer Forums Voice Design Education Voice Design 101 (On Demand Webinar) Intro to ASK (On Demand Webinar) Alexa on Udemy Weekly Office Hours [...]
Sun, 26 Jun 2016 22:00:00 PDTIn June 2015, Amazon Web Services announced that it would launch a new AWS infrastructure region in India. Today, I’m happy to announce that the Asia Pacific (Mumbai) Region is generally available for use by customers worldwide. The opportunity to revolutionize A region in India has been highly sought after by companies around the world who want to participate in one of the most significant economic opportunities in the world – India, a rising economy that holds tremendous promise for growth, a thriving technology hub with a rich eco-system of technology talent, and more. Rapid economic growth in India is creating several business opportunities such as distributed ledger technology with blockchains that could drive efficiencies in the real estate market, Fin-Tech innovations such as P2P mobile apps that have the power to change the social economic lives of people through financial inclusion, applying the sharing economy from cabs to other modes of transportation such as two-wheelers and tractors, telemedicine in the remote reaches of the nation with smartphone apps, or enabling the agricultural sector with on-demand diagnostics to improve farm yield, to name just a few. The platform to revolutionize Market innovators and change agents need a comprehensive infrastructure platform that can reliably scale on-demand. Here are the benefits of a comprehensive platform, with customer examples: A connected platform to sense the business environment Examples of continuous sensing are found in the managed cloud platform built by Rachio on AWS IoT to enable the secure interaction of its connected devices with cloud applications/other devices. In addition, Change Healthcare (previously known as Emdeon) uses Amazon SNS to handle millions of confidential client transactions daily to process claims and pharmacy requests serving over 340K physicians and 60K pharmacies in full compliance with healthcare industry regulations. Seamless ingestion of large volumes of sensed data AdiMap uses Amazon Kinesis to process real-time streaming online ad data and job feeds, and processes them for storage in petabyte-scale Amazon Redshift warehouses to glean business insights for jobs, ad spend, or financials for mobile apps. Advanced problem solving that connects big data with machine learning BuildFax illustrates a practical use case using Amazon Machine Learning to provide roof-age and job-cost estimations for insurers and builders, with property-specific values that don’t need to rely on broad, zip code-level estimates. At-scale computing and visual analysis DNAnexus deploys its customers’ genomic pipelines on Amazon EC2 for highly complex and sensitive DNA research activities. On a more playful note, for those that are inclined to look at our serverless compute architecture, I would love to reacquaint you with Dubsmash’s innovative use of AWS Lambda. A workflow engine to drive business decisions NASA’s Jet Propulsion Laboratory (JPL) used Amazon SWF as an integral part of several missions, including the MER and Carbon in the Arctic Reservoir Vulnerability Experiment (CARVE). NASA/JPL engineers used Amazon SWF and integrated the service with the Polyphony pipelines responsible for data processing of Mars images for tactical operations; expressing it with SWF requires a few simple lines of Java code together with AWS Flow Framework annotations. Let’s build groundbreaking innovations together I hope these short sketches illustrate our optimism in what the future holds. We sincerely believe that such capabilities permit creative expressions for unique solutions that are not only affordable but also scale reliably in order to drive meaningful benefits to the end-user or drive efficiency into business operations. For more details, see the case studies at All AWS Customer Stories. We are excited to offer a compl[...]
Fri, 10 Jun 2016 07:00:00 PDTBuilding your applications with only managed components has become very popular, and AWS Lambda plays a crucial role in that. I see a tremendous interest in examples how to build such applications, and articles such as "The Serverless Start-Up - Down With Servers!" about teletext.io are read eagerly around the globe. If you are looking for more examples there are the Lambda Serverless Reference Architectures that can serve as the blueprint for building your own serverless applications. Mobile Backend Serverless Reference Architecture The Mobile Backend reference architecture demonstrates how to use AWS Lambda along with other services to build a serverless backend for a mobile application. The specific example application provided in this repository enables users to upload photos and notes using Amazon Simple Storage Service (Amazon S3) and Amazon API Gateway respectively. The notes are stored in Amazon DynamoDB, and are processed asynchronously using DynamoDB streams and a Lambda function to add them to an Amazon CloudSearch domain. In addition to the source code for the Lambda functions, this repository also contains a prototype iOS application that provides examples for how to use the AWS Mobile SDK for iOS to interface with the backend resources defined in the architecture. Real-time File Processing Serverless Reference Architecture The Real-time File Processing reference architecture is a general-purpose, event-driven, parallel data processing architecture that uses AWS Lambda. This architecture is ideal for workloads that need more than one data derivative of an object. This simple architecture is described in the "Fanout S3 Event Notifications to Multiple Endpoints" blog post on the AWS Compute Blog. This sample application demonstrates a Markdown conversion application where Lambda is used to convert Markdown files to HTML and plain text. Web Applications Serverless Reference Architecture By combining AWS Lambda with other AWS services, developers can build powerful web applications that automatically scale up and down and run in a highly available configuration across multiple data centers—with zero administrative effort required for scalability, backups, or multi–data center redundancy. This example looks at using AWS Lambda and Amazon API Gateway to build a dynamic voting application, which receives votes via SMS, aggregates the totals into Amazon DynamoDB, and uses Amazon Simple Storage Service (Amazon S3)to display the results in real time. The architecture can be created with an AWS CloudFormation template. The template does the following: Creates an S3 bucket named to hold your web app. Creates a DynamoDB table named VoteApp to store votes Creates a DynamoDB table named VoteAppAggregates to aggregate vote totals Creates a Lambda function that allows your application to receive votes Creates a Lambda function that allows your application to aggregate votes Creates an AWS Identity and Access Management (IAM) role and policy to allow Lambda functions to write to Amazon CloudWatch Logs and write and query the DynamoDB tables IoT Backend Serverless Reference Architecture The Internet of Things (IoT) Backend reference architecture demonstrates how to use AWS Lambda in conjunction with Amazon Kinesis, Amazon DynamoDB, Amazon Simple Storage Service (Amazon S3), and Amazon CloudWatch to build a serverless system for ingesting and processing sensor data. By leveraging these services, you can build cost-efficient applications that can meet the massive scale required for processing the data generated by huge deployments of connected devices. This repository contains sample code for all the Lambda functions depicted in this diagram as well as a AWS CloudFormation template for creating the functions and related resources. There is also a simple webpage that you can run locally[...]
Fri, 11 Mar 2016 06:00:00 PSTThe epoch of AWS is the launch of Amazon S3 on March 14, 2006, now almost 10 years ago. Looking back over the past 10 years, there are hundreds of lessons that we’ve learned about building and operating services that need to be secure, reliable, scalable, with predictable performance at the lowest possible cost. Given that AWS is a pioneer in building and operating these services world-wide, these lessons have been of crucial importance to our business. As we’ve said many times before, “There is no compression algorithm for experience.” With over a million active customers per month, who in turn may serve hundreds of millions of their own customers, there is no lack of opportunities to gain more experience and perhaps no better environment for continuous improvement in the way we serve our customers. I have picked a few of these lessons to share with you in the hope that they may be of use for you as well. 1. Build evolvable systems Almost from day one, we knew that the software we were building would not be the software that would be running a year later. The expectation was that with each order or two of magnitude, we would need to revisit and revise the architecture to make sure we could address the issues of scale. But we couldn’t adopt the old style approach of upgrading systems through a maintenance outage, as many businesses around the world are relying on our platform for 24/7 availability. We needed to build such an architecture that we could introduce new software components without taking the service down. Marvin Theimer, Amazon Distinguished Engineer, once jokingly said that the evolution of Amazon S3 could best be described as starting off as a single engine Cessna plane, but over time the plane was upgraded to a 737, then a group of 747s, all the way to the large fleet of Airbus 380s that it is now. All the while, we were refueling in midair and moving customers from plane to plane without them even realizing it. 2. Expect the unexpected Failures are a given and everything will eventually fail over time: from routers to hard disks, from operating systems to memory units corrupting TCP packets, from transient errors to permanent failures. This is a given, whether you are using the highest quality hardware or lowest cost components. This becomes an even more important lesson at scale: for example, as S3 processes trillions and trillions of storage transactions, anything that has even the slightest probability of error will become realistic. Many of those failure scenarios can be anticipated beforehand, but many more are unknown at design and build time. We needed to build systems that embrace failure as a natural occurrence even if we did not know what the failure might be. Systems need to keep running even if the “house is on fire.” It is important to be able to manage pieces that are impacted without the need to take the overall system down. We’ve developed the fundamental skill of managing the “blast radius” of a failure occurrence such that the overall health of the system can be maintained. 3. Primitives not frameworks Pretty quickly, we started to realize that the way customers would like to use our services was a work in progress. When customers left the constraining, old world of IT hardware and datacenters behind, they started to develop systems with new and interesting usage patterns that no one had ever seen before. As such, we needed to be ultra-agile to make sure we were catering to our customers’ needs. One of the most important mechanisms we provided was to offer customers a collection of primitives and tools, where they could pick and choose their preferred way to engage with the AWS cloud, instead of only providing one framework that they are forced to use, which includes everything and the kitchen sink. This app[...]
Wed, 06 Jan 2016 15:00:00 PSTIn November, Amazon Web Services announced that it would launch a new AWS infrastructure region in South Korea. Today, I’m happy to announce that the Asia Pacific (Seoul) Region is now generally available for use by customers worldwide. A region in South Korea has been highly requested by companies around the world who want to take full advantage of Korea’s world-leading Internet connectivity and provide their customers with quick, low-latency access to websites, mobile applications, games, SaaS applications, and more. We’ve also been hearing many requests from Korean companies, including large enterprises like Samsung and Mirae Asset. For example, Samsung Electronic Printing used AWS to deploy its Printing Apps Center in a way that didn’t require them to invest up-front capital and kept total costs quite low. Mirae Asset Global Investments improved its web service environment and reduced annual management costs by 50% by consolidating the management of all web services, including servers, network, database, and security. We believe that with the launch of the Seoul Region, AWS will enable many more enterprise customers in Korea to reduce the cost of their IT operations and innovate faster in critical new areas such as big data analysis, Internet of Things, and more. Many of these enterprises are assisted by our extensive partner ecosystem in Korea. The rapidly expanding AWS Partner Network (APN) in Korea includes independent software vendors (ISVs) and systems integrators (SIs) who are building innovative solutions and services around the AWS cloud. ISV partners such as Ahnlab, IGAWorks, Hancom, TMAXSoft, and Dreamline are providing a variety of software, security, and connectivity solutions that can be used in conjunction with AWS. SIs such as Vsystems, Bespin Global, Megazone, and GS Neotek are helping enterprises to migrate to AWS, deploy mission-critical applications on AWS, or are providing a full range of monitoring, automation, and management services for customers’ AWS environments. More details on these partners and solutions can be found at https://aws.amazon.com/partners/. The Seoul Region also gives Korean gaming companies the freedom to successfully enable global services. For example, Nexon is Korea’s premier game company, operating 150 games in 150 countries, including major PC games such as FIFA Online 3, MapleStory 2, and Sudden Attack. Nexon uses AWS global infrastructure to manage its IT infrastructure more effectively, and they are now using AWS for their domestic workloads as well. With the Seoul Region now available, Nexon plans to use AWS not just for mobile games but also for latency-sensitive PC online games. All of the top 10 gaming companies in Korea use AWS, and we look forward to continuing to support their global growth and continued success. Finally, the Seoul Region brings the benefits of the cloud much closer to home for Korean startups. In 2015, we expanded the AWS Activate program in Korea to provide startups with the resources needed to get started on AWS, such as access to guidance and 1:1 time with AWS experts as well as web-based training, self-paced labs, customer support, third-party offers, and AWS promotional credits. Through local partnerships with leading venture capitalists (VCs), accelerators, and incubators such as SparkLabs, Primer, Mashup Angels, BonAngels, TheVentures, and Futureplay, 250+ startups in Korea participated in the AWS Activate program this year, and we are excited to see what they are able to achieve with an AWS region in Korea. You can learn more about our growing global infrastructure footprint at http://aws.amazon.com/about-aws/globalinfrastructure. [...]
Thu, 05 Nov 2015 20:00:00 PSTYesterday, AWS evangelist Jeff Barr wrote that AWS will be opening a region in South Korea in early 2016 that will be our 5th region in Asia Pacific. Customers can choose between 11 regions around the world today and, in addition to Korea, we are adding regions in India, a second region in China, and Ohio in 2016. Today, I am excited to add the United Kingdom to that list! The AWS UK region will be our third in the European Union (EU), and we're shooting to have it ready by the end of 2016 (or early 2017). This region will provide even lower latency and strong data sovereignty to local users. More startups, small and medium businesses, large enterprises, universities, and government organizations all over the world are moving to the AWS Cloud faster than ever before. We are committed to meeting our customers’ increasing needs for capacity and for powerful AWS services that eliminate the heavy lifting of the underlying IT infrastructure -- allowing them to focus more of their precious resources on their core business. Leading UK organizations were among the early adopters of the cloud when we first started AWS back in 2006 and we continue to help them drive increased agility, lower IT costs, and easily scale globally. Here are some examples of how our UK customers are using the AWS platform: Hot Startups – Shazam, Hailo, Omnifone, Yplan, SwiftKey, Aire, GoSquared Mid-sized Organisations – Haven Power, Holiday Extras, Exeter Family Friendly, Royal Opera House, Total Jobs, Retail Companies – Shop Direct, Nisa Retail, Kurt Geiger, Sport Pursuit Enterprise Companies – Unilever, ATOC, National Rail Enquiries Media and Entertainment – BBC, Channel 4, ITV, News UK, The FT, Trinity Mirror, The Guardian Public Sector & Not-for-Profit – UCAS, Makewaves, JustGiving The new region, coupled with the existing AWS regions in Dublin and Frankfurt, will provide customers with quick, low-latency access to websites, mobile applications, games, SaaS applications, big data analysis, Internet of Things (IoT) applications, and more. [...]
Wed, 07 Oct 2015 06:00:00 PDTWe live in a world where massive volumes of data are being generated from websites, connected devices and mobile apps. In such a data intensive environment, making key business decisions such as running marketing and sales campaigns, logistic planning, financial analysis, and ad targeting require deriving insights from these data. However, the data infrastructure to collect, store, and process data is geared primarily towards developers and IT professionals (e.g., Amazon Redshift, Amazon DynamoDB, Amazon EMR) whereas insights need to be derived by not just technical professionals but also non-technical, business users. In our quest to enable the best data storage options for customers, over the years we have built several innovative database solutions such as Amazon RDS, Amazon RDS for Aurora, Amazon DynamoDB, and Amazon Redshift. Not surprisingly, customers are using them to collect and store massive amounts of data. Yet, the process of deriving actionable insights out of this wide variety of data sources is not easy. Traditionally, companies had to invest in a lot of complex tools to discover their data sets, ETL tools to prepare for analysis, and separate tools for analyzing and providing visually interactive dashboards. Today, I am excited to share with you a brand new service called Amazon QuickSight that aims to simplify the process of deriving insights from a wide variety of data sources quickly, easily and at a low cost. QuickSight is a very fast, cloud powered, business intelligence service for the 1/10th the cost of old-guard BI solutions. Big data challenges Over the last several years, AWS has delivered on a comprehensive set of services to help customers collect, store, and process their growing volume of data. Today, many thousands of companies—from large enterprises such as Johnson & Johnson, Samsung, and Philips to established technology companies such as Netflix and Adobe to innovative startups such as Airbnb, Yelp, and Foursquare—use Amazon Web Services for their big data needs. Every day, large amount of data is generated from customer applications running on top of AWS infrastructure, collected and streamed using services like Amazon Kinesis, and stored in AWS relational data sources such as Amazon RDS, Amazon Aurora, and Amazon Redshift; NoSQL data sources such as Amazon DynamoDB; and file-based data sources such as Amazon S3. Customers also use a variety of different tools, including Amazon EMR for Hadoop, Amazon Machine Learning, AWS Data Pipeline, and AWS Lambda to process and analyze their data. There’s an inherent gap between the data that is collected, stored, and processed and the key decisions that business users make on a daily basis. Put simply, data is not always readily available and accessible to organizational end users. Most business users continue to struggle answering key business questions such as “Who are my top customers and what are they buying?”, “How is my marketing campaign performing?”, and “Why is my most profitable region not growing?” While BI solutions have existed for decades, customers have told us that it takes an enormous amount of time, IT effort, and money to bridge this gap. Traditional BI solutions typically require teams of data engineers to spend several months building complex data models and synthesizing the data before they can generate their first report. These solutions lack interactive data exploration and visualization capabilities, limiting most business users to canned reports and pre-selected queries. On-premise BI tools also require companies to provision and maintain complex hardware infrastructure and invest in expensive software licenses, maintenance fees, and support fees that cost upwards of t[...]
Mon, 28 Sep 2015 06:00:00 PDTAWS re:Invent is just over one week away—as I prepare to head to Vegas, I’m pumped up about the chance to interact with AWS-powered startups from around the world. One of my favorite parts of the week is being able to host three startup-focused sessions Thursday afternoon: The Startup Scene in 2016: a Visionary Panel [Thursday, 2:45PM] In this session, I’ll moderate a diverse panel of technology experts who’ll discuss emerging trends all startups should be aware of, including how local governments, microeconomic trends, evolving accelerator programs, and the AWS cloud are influencing the global startup scene. This panel will include: Tracy DiNunzio, Founder & CEO, Tradesy Michael DeAngelo, Deputy CIO, State of Washington Ben Whaley, Founder & Principal Consultant, WhaleTech LLC Jason Seats, Managing Director (Austin), & Partner, Techstars CTO-to-CTO Fireside Chat [Thursday, 4:15 PM] This is one of my favorite sessions as I get a chance to sit down and get inside the minds of technical leaders behind some of the most innovative and disruptive startups in the world. I’ll have 1x1 chats with the following CTOs: Laks Srini, CTO and Co-founder, Zenefits Mackenzie Kosut, Head of Technical Operations, Oscar Health Jason MacInnes, CTO, DraftKings Gautam Golwala, CTO and Co-founder, Poshmark 4th Annual Startup Launches [Thursday, 5:30 PM] To wrap up our startup track, in the 4th Annual Startup Launches event we’ll invite five AWS-powered startups to launch their companies on stage, immediately followed by a happy hour. I can’t share the lineup as some of these startups are in stealth mode, but I can promise you this will be an exciting event with each startup sharing a special offer, exclusive to those of you in attendance. Other startup activities Startup Insights from a Venture Capitalists Perspective [Thursday, 1:30 PM] Immediately before I take the stage, you can join a group of venture capitalists as they share insights and observations about the global startup ecosystem: each panelist will share the most significant insight they’ve gained in the past 12 months and what they believe will be the most impactful development in the coming year. The AWS Startup Pavilion [Tuesday – Thursday] If you’re not able to join the startup sessions Thursday afternoon, I encourage you to swing by the AWS Startup Pavilion (within re:Invent Central, booth 1062) where you can meet the AWS startup team, mingle with other startups, chat 1:1 with an AWS architect, and learn about AWS Activate. Startup Stop on the re:Invent Pub Crawl [Wednesday evening] And to relax and unwind in the evening, you won’t want to miss the startup stop on the re:Invent pub crawl, at the Rockhouse within The Grand Canal Shoppes at The Venetian. This is the place to be for free food, drinks, and networking during the Wednesday night re:Invent pub crawl. Look forward to seeing you in Vegas! [...]
Tue, 08 Sep 2015 06:00:00 PDTAmazon Web Services (AWS) has been working closely with the startup community in London, and Europe, since we launched back in 2006. We have grown substantially in that time and today more than two thirds of the UK’s startups with valuations of over a billion dollars, including Skyscanner, JustEat, Powa, Fanduel and Shazam, are all leveraging our platform to deliver innovative services to customers around the world. This week I will have the pleasure of meeting up with our startup customers to we celebrate the opening of the first of the AWS Pop-up Lofts to open outside of the US in one of the greatest cities in the World, London. The London Loft opening will be followed in quick succession by our fourth Pop-up Loft opening its doors in Berlin. Both London and Berlin are vibrant cities with a concentration of innovative startups building their businesses on AWS. The Loft’s will give them a physical place to not only learn about our services but will aim to help cultivate a community of AWS customers that can learn from each other. Every time I’ve visited the Loft’s in both San Francisco and New York there has been a great buzz with people getting advice from our solution architects, getting training or attending talks and demos. By opening the London and Berlin Loft’s we’re hoping to cultivate that same community and expand on the base of loyal startups we have, such as Hailo, YPlan, SwiftKey, Mendley, GoSquared, Playmob and Yoyo Wallet, to help them to grow their companies globally and be successful. You can expect to see some of the brightest and most creative minds in the industry being on hand in the Lofts to help and I’d encourage all local startups to make the most of the resources which will be at your fingertips, ranging from technology resources through access to our vast network of customers, partners, accelerators, incubators and venture capitalists who will all be in the loft to help you gain the insight you need and provide advice on how to secure funding, and gain the ‘softer skills’ needed to to grow your businesses. The AWS Pop-up Loft, in London will be open from September 10 to October 29 between 10am and 6pm and later for evening events, Monday through Friday, in Moorgate. You can go online now at http://awsloft.london, to make one-on-one appointments with an AWS expert, register for boot camps and technical sessions, including: Ask an Architect: an hour session which can be scheduled with a member of the AWS technical team. Bring your questions about AWS architecture, cost optimisation, services and features, or anything else AWS related. You can also drop in if you don’t have an appointment. Technical Bootcamps: a one-day training sessions, taught by experienced AWS instructors and solutions architects. You will get hands-on experience using a live environment with the AWS Management Console. There is a ‘Getting started with AWS’ bootcamp on Chef bootcamp which will show customers how they can safeguard their infrastructure, manage complexity, and accelerate time to market. Self-paced Hands-on Labs: beginners through advanced users can attend the labs which will help sharpen AWS technical skills at a personal pace and are available for free in the Loft during operating hours. The London Loft will also feature an IoT Lab with a range of devices running on AWS services, many of which have been developed by our Solutions Architects. Visitors to the Loft will be able to participate in live demos and Q&A opportunities, as our technical team demonstrates what is possible with IoT on AWS. You are all invited to join us for the grand opening party at the Loft in London on September 10 at [...]
Thu, 20 Aug 2015 06:00:00 PDTToday, we are releasing a plugin that allows customers to use the Titan graph engine with Amazon DynamoDB as the backend storage layer. It opens up the possibility to enjoy the value that graph databases bring to relationship-centric use cases, without worrying about managing the underlying storage. The importance of relationships Relationships are a fundamental aspect of both the physical and virtual worlds. Modern applications need to quickly navigate connections in the physical world of people, cities, and public transit stations as well as the virtual world of search terms, social posts, and genetic code, for example. Developers need efficient methods to store, traverse, and query these relationships. Social media apps navigate relationships between friends, photos, videos, pages, and followers. In supply chain management, connections between airports, warehouses, and retail aisles are critical for cost and time optimization. Similarly, relationships are essential in many other use cases such as financial modeling, risk analysis, genome research, search, gaming, and others. Traditionally, these connections have been stored in relational databases, with each object type requiring its own table. When using relational databases, traversing relationships requires expensive table JOIN operations, causing significantly increased latency as table size and query complexity grow. Enter graph databases Graph databases belong to the NoSQL family, and are optimized for storing and traversing relationships. A graph consists of vertices, edges, and associated properties. Each vertex contains a list of properties and edges, which represent the relationships to other vertices. This structure is optimized for fast relationship query and traversal, without requiring expensive table JOIN operations. In this way, graphs can scale to billions of vertices and edges, while allowing efficient queries and traversal of any subset of the graph with consistent low latency that doesn’t grow proportionally to the overall graph size. This is an important benefit for many use cases that involve accessing and traversing small subsets of a large graph. A concrete example is generating a product recommendation based on purchase interests of a user’s friends, where the relevant social connections are a small subset of the total network. Another example is for tracking inventory in a vast logistics system, where only a subset of its locations is relevant for a specific item. For us at Amazon, the challenge of tracking inventory at massive scale is not just theoretical, but very real. Graph databases at Amazon Like many AWS innovations, the desire to build a solution for a scalable graph database came from Amazon’s retail business. Amazon runs one of the largest fulfillment networks in the world, and we need to optimize our systems to quickly and accurately track the movement of vast amounts of inventory. This requires a database that can quickly traverse the logistics history for a given item or order. Graph databases are ideal for the task, since they make it easy to store and retrieve each item’s logistics history. Our criteria for choosing the right graph engine were: The ability to support a graph containing billions of vertices and edges. The ability to scale with the accelerating pace of new items added to the catalog, and new objects and locations in the company’s expanding fulfillment network. After evaluating different technologies, we decided to use Titan, a distributed graph database engine optimized for creating and querying large graphs. Titan has a pluggable storage architecture, using existing NoSQL databases as[...]
Mon, 20 Jul 2015 06:00:00 PDTIn my last post about Amazon EC2 Container Service (Amazon ECS), I discussed the two key components of running modern distributed applications on a cluster: reliable state management and flexible scheduling. Amazon ECS makes building and running containerized applications simple, but how that happens is what makes Amazon ECS interesting. Today, I want to explore the Amazon ECS architecture and what this architecture enables. Below is a diagram of the basic components of Amazon ECS: How we coordinate the cluster Let’s talk about what Amazon ECS is actually doing. The core of Amazon ECS is the cluster manager, a backend service that handles the tasks of cluster coordination and state management. On top of the cluster manager sits various schedulers. Cluster management and container scheduling are components decoupled from each other allowing customers to use and build their own schedulers. A cluster is just a pool of compute resources available to a customer’s applications. The pool of resources, at this time, is the CPU, memory, and networking resources of Amazon EC2 instances as partitioned by containers. Amazon ECS coordinates the cluster through the Amazon ECS Container Agent running on each EC2 instance in the cluster. The agent allows Amazon ECS to communicate with the EC2 instances in the cluster to start, stop, and monitor containers as requested by a user or scheduler. The agent is written in Go, has a minimal footprint, and is available on GitHub under an Apache license. We encourage contributions and feedback is most welcome. How we manage state To coordinate the cluster, we need to have a single source of truth on the clusters themselves: EC2 instances in the clusters, tasks running on the EC2 instances, containers that make up a task, and resources available or occupied (e.g., networks ports, memory, CPU, etc). There is no way to successfully start and stop containers without an accurate knowledge of the state of the cluster. In order to solve this, state needs to be stored somewhere, so at the heart of any modern cluster manager is a key/value store. This key/value store acts as the single source of truth for all information on the cluster (state, and all changes to state transitions) are entered and stored here. To be robust and scalable, this key/value store needs to be distributed for durability and availability, to protect against network partitions or hardware failures. But because the key/value store is distributed, making sure data is consistent and handling concurrent changes becomes more difficult, especially in an environment where state constantly changes (e.g., containers stopping and starting). As such, some form of concurrency control has to be put in place in order to make sure that multiple state changes don’t conflict. For example, if two developers request all the remaining memory resources from a certain EC2 instance for their container, only one container can actually receive those resources and the other would have to be told their request could not be completed. To achieve concurrency control, we implemented Amazon ECS using one of Amazon’s core distributed systems primitives: a Paxos-based transactional journal based data store that keeps a record of every change made to a data entry. Any write to the data store is committed as a transaction in the journal with a specific order-based ID. The current value in a data store is the sum of all transactions made as recorded by the journal. Any read from the data store is only a snapshot in time of the journal. For a write to succeed, the write proposed must be the latest transaction since the las[...]
Fri, 17 Jul 2015 10:00:00 PDT
Data compression today is still as important as it was in the early days of computing. Although in those days all computer and storage resources were very limited, the objects in use were much smaller than today. We have seen a shift from generic compression to compression for specific file types, especially those in images, audio and video. In this weekend's back to basic reading we go back in time, 1987 to be specific, when Leweler and Hirschberg wrote a survey paper that covers the 40 years of data compression research. It covers all the areas that we like in a back to basics paper, it does not present the most modern results but it gives you a great understanding of the fundamentals. It is a substantial paper but easy to read.
Data compression, D.A. Lelewer and D.S. Hirschberg, Data compression, Computing Surveys 19,3 (1987) 261-297.
Tue, 14 Jul 2015 10:00:00 PDTIn just three short years, Amazon DynamoDB has emerged as the backbone for many powerful Internet applications such as AdRoll, Druva, DeviceScape, and Battlecamp. Many happy developers are using DynamoDB to handle trillions of requests every day. I am excited to share with you that today we are expanding DynamoDB with streams, cross-region replication, and database triggers. In this blog post, I will explain how these three new capabilities empower you to build applications with distributed systems architecture and create responsive, reliable, and high-performance applications using DynamoDB that work at any scale. DynamoDB Streams enables your application to get real-time notifications of your tables’ item-level changes. Streams provide you with the underlying infrastructure to create new applications, such as continuously updated free-text search indexes, caches, or other creative extensions requiring up-to-date table changes. DynamoDB Streams is the enabling technology behind two other features announced today: cross-region replication maintains identical copies of DynamoDB tables across AWS regions with push-button ease, and triggers execute AWS Lambda functions on streams, allowing you to respond to changing data conditions. Let me expand on each one of them. DynamoDB Streams DynamoDB Streams provides you with a time-ordered sequence, or change log, of all item-level changes made to any DynamoDB table. The stream is exposed via the familiar Amazon Kinesis interface. Using streams, you can apply the changes to a full-text search data store such as Elasticsearch, push incremental backups to Amazon S3, or maintain an up-to-date read cache. I have heard from many of you that one of the common challenges you have is keeping DynamoDB data in sync with other data sources, such as search indexes or data warehouses. In traditional database architectures, database engines often run a small search engine or data warehouse engines on the same hardware as the database. However, the model of collocating all engines in a single database turns out to be cumbersome because the scaling characteristics of a transactional database are different from those of a search index or data warehouse. A more scalable option is to decouple these systems and build a pipe that connects these engines and feeds all change records from the source database to the data warehouse (e.g., Amazon Redshift) and Elasticsearch machines. The velocity and variety of data that you are managing continues to increase, making your task of keeping up with the change more challenging as you want to manage the systems and applications in real time and respond to changing conditions. A common design pattern is to capture transactional and operational data (such as logs) that require high throughput and performance in DynamoDB, and provide periodic updates to search clusters and data warehouses. However, in the past, you had to write code to manage the data changes and deal with keeping the search engine and data warehousing engines in sync. For cost and manageability reasons, some developers have collocated the extract job, the search cluster, and data warehouses on the same box, leading to performance and scalability compromises. DynamoDB Streams simplifies and improves this design pattern with a distributed systems approach. You can enable the DynamoDB Streams feature for a table with just a few clicks using the AWS Management Console, or you can use the DynamoDB API. Once configured, you can use an Amazon EC2 instance to read the stream using the Amazon Kinesis interface, an[...]