Subscribe: All Things Distributed
http://www.allthingsdistributed.com/index.xml
Added By: Feedage Forager Feedage Grade A rated
Language: English
Tags:
amazon  applications  availability  aws  business  cloud  companies  customers  data  database  new  region  service  time  today 
Rate this Feed
Rate this feedRate this feedRate this feedRate this feedRate this feed
Rate this feed 1 starRate this feed 2 starRate this feed 3 starRate this feed 4 starRate this feed 5 star

Comments (0)

Feed Details and Statistics Feed Statistics
Preview: All Things Distributed

All Things Distributed



Werner Vogels' weblog on building scalable and robust distributed systems.



Last Build Date: Wed, 11 Apr 2018 07:50:52 PDT

Copyright: Copyright 2011
 



Changing the calculus of containers in the cloud

Wed, 11 Apr 2018 08:00:00 PDT

I wrote to you over two years ago about what happens under the hood of Amazon ECS. Last year at re:Invent, we launched AWS Fargate, and today, I want to explore how Fargate fundamentally changes the landscape of container technology. I spend a lot of time talking to our customers and leaders at Amazon about innovation. One of the things I've noticed is that ideas and technologies which dramatically change the way we do things are rarely new. They're often the combination of an existing concept with an approach, technology, or capability in a particular way that's never been successfully tried before. The rapid embrace of containers in the past four years is the result of blending old technology (containers) with a new toolchain and workflow (i.e., Docker), and the cloud. In our industry, four years is a long time, but I think we've only just started exploring how this combination of code packaging, well-designed workflows, and the cloud can reshape the ability of developers to quickly build applications and innovate. Containers solve a fundamental code portability problem and enable new infrastructure patterns on the cloud. Having a consistent, immutable unit of deployment to work with lets you abstract away all the complexities of configuring your servers and deployment pipelines every time you change your code or want to run your app in a different place. But containers also put another layer between your code and where it runs. They are an important, but incremental, step on the journey of being able to write code and have it run in the right place, with the right scale, with the right connections to other bits of code, and the right security and access controls. Solving these higher order problems of deploying, scheduling, and connecting containers across environments gave us container management tools. Container orchestration has always seemed to me to be very not cloud native. Managing a large server cluster and optimizing the scheduling of containers all backed by a complex distributed state store is counter to the premise of the cloud. Customers choose the cloud to pay as they go, not have to guess capacity, get deep operational control without operational burden, build loosely coupled services with limited blast radii to prevent failures, and self-service for everything they need to run their code. You should be able to write your code and have it run, without having to worry about configuring complex management tools, open source or not. This is the vision behind AWS Fargate. With Fargate, you don't need to stand up a control plane, choose the right instance type, or configure all the other components of your application stack like networking, scaling, service discovery, load balancing, security groups, permissions, or secrets management. You simply build your container image, define how and where you want it to run, and pay for the resources you need. Fargate has native integrations to Amazon VPC, Auto Scaling, Elastic Load Balancing, IAM roles, and Secrets Management. We've taken the time to make Fargate production ready with a 99.99% uptime SLA and compliance with PCI, SOC, ISO, and HIPAA. With AWS Fargate, you can provision resources to run your containers at a much finer grain than with an EC2 instance. You can select exactly the CPU and memory your code needs and the amount you pay scales exactly with how many containers you run. You don't have to guess at capacity to handle spikey traffic, and you get the benefit of perfect scale, which lets you offload a ton of operational effort onto the cloud. MiB for MiB, this might mean that cloud native technologies like Fargate look more expensive than more traditional VM infrastructure on paper. But if you look at the full cost of running an app, we believe most applications will be cheaper with Fargate as you only pay what you need. Our customers running Fargate see big savings in the developer hours required to keep their apps running smoothly. The entire ecosystem of container orchestration solutions arose out of necessity [...]



Looking back at 10 years of compartmentalization at AWS

Mon, 26 Mar 2018 10:00:00 PDT

At AWS, we don't mark many anniversaries. But every year when March 14th comes around, it's a good reminder that Amazon S3 originally launched on Pi Day, March 14, 2006. The Amazon S3 team still celebrate with homemade pies! March 26, 2008 doesn't have any delicious desserts associated with it, but that's the day when we launched Availability Zones for Amazon EC2. A concept that has changed infrastructure architecture is now at the core of both AWS and customer reliability and operations. Powering the virtual instances and other resources that make up the AWS Cloud are real physical data centers with AWS servers in them. Each data center is highly reliable, and has redundant power, including UPS and generators. Even though the network design for each data center is massively redundant, interruptions can still occur. Availability Zones draw a hard line around the scope and magnitude of those interruptions. No two zones are allowed to share low-level core dependencies, such as power supply or a core network. Different zones can't even be in the same building, although sometimes they are large enough that a single zone spans several buildings. We launched with three autonomous Availability Zones in our US East (N. Virginia) Region. By using zones, and failover mechanisms such as Elastic IP addresses and Elastic Load Balancing, you can provision your infrastructure with redundancy in mind. When two instances are in different zones, and one suffers from a low-level interruption, the other instance should be unaffected. How Availability Zones have changed over the years Availability Zones were originally designed for physical redundancy, but over time they have become re-used for more and more purposes. Zones impact how we build, deploy, and operate software, as well as how we enforce security controls between our largest systems. For example, many AWS services are now built so that as much functionality as possible can be autonomous within an Availability Zone. The calls used to launch and manage EC2 instances, fail over an RDS instance, or handle the health of instances behind a load balancer, all work within one zone. This design has a double benefit. First, if an Availability Zone does lose power or connectivity, the remaining zones are unaffected. The second benefit is even more powerful: if there is an error in the software, the risk of that error affecting other zones is minimized. We maximize this benefit when we deploy new versions of our software, or operational changes such as a configuration edit, as we often do so zone-by-zone, one zone in a Region at a time. Although we automate, and don't manage instances by hand, our developers and operators know not to build tools or procedures that could impact multiple Availability Zones. I'd wager that every new AWS engineer knows within their first week, if not their first day, that we never want to touch more than one zone at a time. Availability Zones run deep in our AWS development and operations culture, at every level. AWS customers can think of zones in terms of redundancy, "Use two or more Availability Zones for reliability." At AWS, we think of zones in terms of isolation, "Stay within the Availability Zone, as much as possible." Silo your traffic or not – you choose When your architecture does stay within an Availability Zone as much as possible, there are more benefits. One is that the latency within a zone is incredibly fast. Today, packets between EC2 instances in the same zone take just tens of microseconds to reach other. Another benefit is that redundant zonal architectures are easier to recover from complex issues and emergent behaviors. If all of the calls between the various layers of a service stay within one Availability Zone, then when issues occur they can quickly be remediated by removing the entire zone from service, without needing to identify the layer or component that was the trigger. Many of you also use this kind of "silo" pattern in y[...]



Infinitely scalable machine learning with Amazon SageMaker

Mon, 19 Mar 2018 09:00:00 PDT

In machine learning, more is usually more. For example, training on more data means more accurate models. At AWS, we continue to strive to enable builders to build cutting-edge technologies faster in a secure, reliable, and scalable fashion. Machine learning is one such transformational technology that is top of mind not only for CIOs and CEOs, but also developers and data scientists. Last re:Invent, to make the problem of authoring, training, and hosting ML models easier, faster, and more reliable, we launched Amazon SageMaker. Now, thousands of customers are trying Amazon SageMaker and building ML models on top of their data lakes in AWS. While building Amazon SageMaker and applying it for large-scale machine learning problems, we realized that scalability is one of the key aspects that we need to focus on. So, when designing Amazon SageMaker we took on a challenge: to build machine learning algorithms that can handle an infinite amount of data. What does that even mean though? Clearly, no customer has an infinite amount of data. Nevertheless, for many customers, the amount of data that they have is indistinguishable from infinite. Bill Simmons, CTO of Dataxu, states, "We process 3 million ad requests a second - 100,000 features per request. That's 250 trillion ad requests per day. Not your run-of-the-mill data science problem!" For these customers and many more, the notion of "the data" does not exist. It's not static. Data always keeps being accrued. Their answer to the question "how much data do you have?" is "how much can you handle?" To make things even more challenging, a system that can handle a single large training job is not nearly good enough if training jobs are slow or expensive. Machine learning models are usually trained tens or hundreds of times. During development, many different versions of the eventual training job are run. Then, to choose the best hyperparameters, many training jobs are run simultaneously with slightly different configurations. Finally, re-training is performed every x-many minutes/hours/days to keep the models updated with new data. In fraud or abuse prevention applications, models often need to react to new patterns in minutes or even seconds! To that end, Amazon SageMaker offers algorithms that train on indistinguishable-from-infinite amounts of data both quickly and cheaply. This sounds like a pipe dream. Nevertheless, this is exactly what we set out to do. This post lifts the veil on some of the scientific, system design, and engineering decisions we made along the way. Streaming algorithms To handle unbounded amounts of data, our algorithms adopt a streaming computational model. In the streaming model, the algorithm only passes over the dataset one time and assumes a fixed-memory footprint. This memory restriction precludes basic operations like storing the data in memory, random access to individual records, shuffling the data, reading through the data several times, etc. Streaming algorithms are infinitely scalable in the sense that they can consume any amount of data. The cost of adding more data points is independent of the entire corpus size. In other words, processing the 10th gigabyte and 1000th gigabyte is conceptually the same. The memory footprint of the algorithms is fixed and it is therefore guaranteed not to run out of memory (and crash) as the data grows. The compute cost and training time depend linearly on the data size. Training on twice as much data costs twice as much and take twice as long. Finally, traditional machine learning algorithms usually consume data from persistent data sources such as local disk, Amazon S3, or Amazon EBS. Streaming algorithms also natively consume ephemeral data sources such as Amazon Kinesis streams, pipes, database query results, and almost any other data source. Another significant advantage of streaming algorithms is the notion of a state. The algorithm state contains all the variables, statistics, and data struc[...]



Unlocking Enterprise systems using voice

Mon, 12 Mar 2018 06:00:00 PDT

At Amazon, we are heavily invested in machine learning (ML), and are developing new tools to help developers quickly and easily build, train, and deploy ML models. The power of ML is in its ability to unlock a new set of capabilities that create value for consumers and businesses. A great example of this is the way we are using ML to deal with one of the world's biggest and most tangled datasets: human speech. Voice-driven conversation has always been the most natural way for us to communicate. Conversations are personal and they convey context, which helps us to understand each other. Conversations continue over time, and develop history, which in turn builds richer context. The challenge was that technology wasn't capable of processing real human conversation. The interfaces to our digital system have been dictated by the capabilities of our computer systems—keyboards, mice, graphical interfaces, remotes, and touch screens. Touch made things easier; it let us tap on screens to get the app that we wanted. But what if touch isn't possible or practical? Even when it is, the proliferation of apps has created a sort of "app fatigue". This essentially forces us to hunt for the app that we need, and often results in us not using many of the apps that we already have. None of these approaches are particularly natural. As a result, they fail to deliver a truly seamless and customer-centric experience that integrates our digital systems into our analog lives. Voice becomes a game changer Using your voice is powerful because it's spontaneous, intuitive, and enables you to interact with technology in the most natural way possible. It may well be considered the universal user interface. When you use your voice, you don't need to adapt and learn a new user interface. Voice interfaces don't need to be application-centric, so you don't have to find an app to accomplish the task that you want. All of these benefits make voice a game changer for interacting with all kinds of digital systems. Until 2-3 years ago we did not have the capabilities to process voice at scale and in real time. The availability of large scale voice training data, the advances made in software with processing engines such as Caffe, MXNet and Tensorflow, and the rise of massively parallel compute engines with low-latency memory access, such as the Amazon EC2 P3 instances have made voice processing at scale a reality. Today, the power of voice is most commonly used in the home or in cars to do things like play music, shop, control smart home features, and get directions. A variety of digital assistants are playing a big role here. When we released Amazon Alexa, our intelligent, cloud-based voice service, we built its voice technology on the AWS Natural Language Processing platform powered by ML algorithms. Alexa is constantly learning, and she has tens of thousands of skills that extend beyond the consumer space. But by using the stickiness of voice, we think there are even more scenarios that can be unlocked at work. Helping more people and organizations use voice People interact with many different applications and systems at work. So why aren't voice interfaces being used to enable these scenarios? One impediment is the ability to manage voice-controlled interactions and devices at scale, and we are working to address this with Alexa for Business. Alexa for Business helps companies voice-enable their spaces, corporate applications, people, and customers. To use voice in the workplace, you really need three things. The first is a management layer, which is where Alexa for Business plays. Second, you need a set of APIs to integrate with your IT apps and infrastructure, and third is having voice-enabled devices everywhere. Voice interfaces are a paradigm shift, and we've worked to remove the heavy lifting associated with integrating Alexa voice capabilities into more devices. For example, Alexa Voice Service (AVS), a cloud-b[...]



Rethinking the 'production' of data

Wed, 20 Dec 2017 09:00:00 PST

This article titled "Daten müssen strategischer Teil des Geschäfts werden" appeared in German last week in the "IT und Datenproduktion" column of Wirtschaftwoche. How companies can use ideas from mass production to create business with data Strategically, IT doesn't matter. That was the provocative thesis of a much-talked-about article from 2003 in the Harvard Business Review by the US publicist Nicolas Carr. Back then, companies spent more than half of their entire investment for their IT, in a non-differentiating way. In a world in which tools are equally accessible for every company, they wouldn't offer any competitive advantage – so went the argument. The author recommended steering investments toward strategically relevant resources instead. In the years that followed, many companies outsourced their IT activities because they no longer regarded them as being part of the core business. A new age Nearly 15 years later, the situation has changed. In today's era of global digitalization there are many examples that show that IT does matter. Developments like cloud computing, the internet of things, artificial intelligence, and machine learning are proving that IT has (again) become a strategic business driver. This is transforming the way companies offer products and services to their customers today. Take the example of industrial manufacturing: in prototyping, drafts for technologically complex products are no longer physically produced; rather, their characteristics can be tested in a purely virtual fashion at every location across the globe by using simulations. The German startup SimScale makes use of this trend. The founders had noticed that in many companies, product designers worked in a very detached manner from the rest of production. The SimScale platform can be accessed through a normal web browser. In this way, designers are part of an ecosystem in which the functionalities of simulations, data and people come together, enabling them to develop better products faster. Value-added services are also playing an increasingly important role for both companies and their customers. For example, Kärcher, the maker of cleaning technologies, manages its entire fleet through the cloud solution "Kärcher Fleet". This transmits data from the company's cleaning devices e.g. about the status of maintenance and loading, when the machines are used, and where the machines are located. The benefit for customers: Authorized users can view this data and therefore manage their inventories across different sites, making the maintenance processes much more efficient. Kärcher benefits as well: By developing this service, the company gets exact insight into how the machines are actually used by its customers. By knowing this, Kärcher can generate new top-line revenue in the form of subscription models for its analysis portal. More than mere support These examples underline that the purpose of software today is not solely to support business processes, but that software solutions have broadly become an essential element in multiple business areas. This starts with integrated platforms that can manage all activities, from market research to production to logistics. Today, IT is the foundation of digital business models, and therefore has a value-added role in and of itself. That can be seen when sales people, for example, interact with their customers in online shops or via mobile apps. Marketers use big data and artificial intelligence to find out more about the future needs of their customers. Breuninger, a fashion department store chain steeped in tradition, has recognized this and relies on a self-developed e-commerce platform in the AWS Cloud. Breuninger uses modern templates for software development, such as Self-Contained Systems (SCS), so that it can increase the speed of software development with agile and autonomous teams and quickly test new features. Each team acts accord[...]



'Paris s'éveille'! Introducing the AWS EU (Paris) Region

Mon, 18 Dec 2017 20:00:00 PST

Today, I'm happy to announce that the AWS EU (Paris) Region, our 18th technology infrastructure Region globally, is now generally available for use by customers worldwide. With this launch, AWS now provides 49 Availability Zones, with another 12 Availability Zones and four Regions in Bahrain, Hong Kong, Sweden, and a second AWS GovCloud (US) Region expected to come online by early 2019. In France, you can find one of the most vibrant startup ecosystems in the world, a strong research community, excellent energy, telecom, and transportation infrastructure, a very strong agriculture and food industry, and some of the most influential luxury brands in the world. The cloud is an opportunity to stay competitive in each of these domains by giving companies freedom to innovate quickly. This is why tens of thousands of French customers already use AWS in Regions around the world. Starting today, developers, startups, and enterprises, as well as government, education, and non-profit organizations can leverage AWS to run applications and store data in France. French companies are using AWS to innovate in a secure way across industries as diverse as energy, financial services, manufacturing, media, pharmaceuticals and health sciences, retail, and more. Companies of all sizes across France are also using AWS to innovate and grow, from startups like AlloResto, CaptainDash, Datadome, Drivy, Predicsis, Payplug, and Silkke to enterprises like Decathlon, Nexity, Soitec, TF1 as well as more than 80 percent of companies listed on the CAC 40, like Schneider Electric, Societe Generale, and Veolia. We are also seeing a strong adoption of AWS within the public sector with organizations using AWS to transform the services they deliver to the citizens of France.Kartable, Les Restos du Coeur, OpenClassrooms, Radio France, SNCF, and many more are using AWS to lower costs and speed up their rate of experimentation so they can deliver reliable, secure, and innovative services to people across the country. The opening of the AWS EU (Paris) Region adds to our continued investment in France. Over the last 11 years, AWS has expanded its physical presence in the country, opening an office in La Defense and launching Edge Network Locations in Paris and Marseille. Now, we're opening an infrastructure Region with three Availability Zones. We decided to locate the AWS data centers in the area of Paris, the capital and economic center of France because it is home to many of the world's largest companies, the majority of the French public sector, and some of Europe's most dynamic startups. To give customers the best experience when connecting to the new Region, today we are also announcing the availability of AWS Direct Connect. Today, customers can connect to the AWS EU (Paris) Region via Telehouse Voltaire. In January 2018, customers will be able to connect via Equinix Paris in January and later in the year via Interxion Paris. Customers that have equipment within these facilities can use Direct Connect to optimize their connection to AWS. In addition to physical investments, we have also continually invested in people in France. For many years, we have been growing teams of account managers, solutions architects, trainers, business development, and professional services specialists, as well as other job functions. These teams are helping customers and partners of all sizes, including systems integrators and ISVs, to move to the cloud. We have also been investing in helping to grow the entire French IT community with training, education, and certification programs. To continue this trend, we recently announced plans for AWS to train, at no charge, more than 25,000 people in France, helping them to develop highly sought-after skills. These people will be granted access to AWS training resources in France via existing programs such as AWS Academy, AWS Educate, AWSome days. They also get access to webi[...]



Expanding the AWS Cloud: Introducing the AWS China (Ningxia) Region

Mon, 11 Dec 2017 20:00:00 PST

(image)

Today, I am happy to announce the general availability of AWS China (Ningxia) Region, operated by Ningxia Western Cloud Data Technology Co. Ltd. (NWCD). This is our 17th Region globally, and the second in China. To comply with China's legal and regulatory requirements, AWS has formed a strategic technology collaboration with NWCD to operate and provide services from the AWS China (Ningxia) Region. Founded in 2015, NWCD is a licensed data center and cloud services provider, based in Ningxia, China.

Coupled with the AWS China (Beijing) Region operated by Sinnet, the AWS China (Ningxia) Region, operated by NWCD, serves as the foundation for new cloud initiatives in China, especially in Western China. Both Regions are helping to transform businesses, increase innovation, and enhance the regional economy.

Thousands of customers in China are already using AWS services operated by Sinnet, to innovate in diverse areas such as energy, education, manufacturing, home security, mobile and internet platforms, CRM solutions, and the dairy industry, among others. These customers include large Chinese enterprises such as Envision Energy, Xiaomi, Lenovo, OPPO, TCL, Hisense, Mango TV, and Mengniu; well-known, fast growing startups including iQiyi, VIPKID, musical.ly, Xiaohongshu, Meitu, and Kunlun; and multinationals such as Samsung, Adobe, ThermoFisher Scientific, Dassault Systemes, Mapbox, Glu, and Ayla Networks. With AWS, Chinese customers can leverage world-class technologies both within China and around the world.

As this breadth of customers shows, we believe that AWS can and will serve China's innovation agenda. We are excited to collaborate with NWCD in Ningxia and Sinnet in Beijing to offer a robust portfolio of services. Our offerings range from our foundational service stack for compute, storage, and networking to our more advanced solutions and applications.

Starting today, China-based developers, startups, and enterprises, as well as government, education, and non-profit organizations, can use AWS to run their applications and store their data in the new AWS China (Ningxia) Region, operated by NWCD. Customers already using the AWS China (Beijing) Region, operated by Sinnet, can select the AWS China (Ningxia) Region directly from the AWS Management Console. New customers can request an account at www.amazonaws.cnto begin using both AWS China Regions.




Accelerate Machine Learning with Amazon SageMaker

Wed, 29 Nov 2017 10:00:00 PST

Applications based on machine learning (ML) can provide tremendous business value. However, many developers find them difficult to build and deploy. As there are few individuals with this expertise, an easier process presents a significant opportunity for companies who want to accelerate their ML usage. Though the AWS Cloud gives you access to the storage and processing power required for ML, the process for building, training, and deploying ML models has unique challenges that often block successful use of this powerful new technology. The challenges begin with collecting, cleaning, and formatting training data. After the dataset is created, you must scale the processing to handle the data, which can often be a blocker. After this, there is often a long process of training that includes tuning the knobs and levers, called hyperparameters, that control the different aspects of the training algorithm. Finally, figuring out how to move the model into a scalable production environment can often be slow and inefficient for those that do not do it routinely. At Amazon Web Services, we've committed to helping you unlock the value of your data through ML, through a set of supporting tools and resources that improve the ML model development experience. From the Deep Learning AMI and the distributed Deep Learning AWS CloudFormation template, to Gluon in Apache MXNet, we've focused on improvements that remove the roadblocks to development. We also recently announced the Amazon ML Solutions Lab, which is a program to help you accelerate your use of ML in products and processes. As the adoption of these technologies continues to grow, customers have demanded a managed service for ML, to make it easier to get started. Today, we are announcing the general availability of Amazon SageMaker. This new managed service enables data scientists and developers to quickly and easily build, train, and deploy ML models without getting mired in the challenges that slow this process down today. Amazon SageMaker provides the following features: Hosted Jupyter notebooks that require no setup, so that you can start processing your training dataset and developing your algorithms immediately. One-click, on-demand distributed training that sets up and tears down the cluster after training. Built-in, high-performance ML algorithms, re-engineered for greater, speed, accuracy, and data-throughput. Built-in model tuning (hyperparameter optimization) that can automatically adjust hundreds of different combinations of algorithm parameters. An elastic, secure, and scalable environment to host your models, with one-click deployment. In the hosted notebook environment, Amazon SageMaker takes care of establishing secure network connections in your VPC and launching an ML instance. This development workspace also comes pre-loaded with the necessary Python libraries and CUDA drivers, attaches an Amazon EBS volume to automatically persist notebook files, and installs TensorFlow, Apache MXNet, and Keras deep learning frameworks. Amazon SageMaker also includes common examples to help you get started quickly. For training, you simply indicate the type and quantity of ML instances you need and initiate training with a single click. Amazon SageMaker then sets up the distributed compute cluster, installs the software, performs the training, and tears down the cluster when complete. You only pay for the resources that you use and never have to worry about the underlying infrastructure. Amazon SageMaker also reduces the amount of time spent tuning models using built-in hyperparameter optimization. This technology automatically adjusts hundreds of different combinations of parameters, to quickly arrive at the best solution for your ML problem. With high-performance algorithms, distributed computing, managed infrastructure, and hyperparameter optimization, Amazon S[...]



Scaling Amazon ElastiCache for Redis with Online Cluster Resizing

Tue, 21 Nov 2017 10:00:00 PST

Amazon ElastiCache embodies much of what makes fast data a reality for customers looking to process high volume data at incredible rates, faster than traditional databases can manage. Developers love the performance, simplicity, and in-memory capabilities of Redis, making it among the most popular NoSQL key-value stores. Redis's microsecond latency has made it a de facto choice for caching. Its support for advanced data structures (for example, lists, sets, and sorted sets) also enables a variety of in-memory use cases such as leaderboards, in-memory analytics, messaging, and more. Four years ago, as part of our AWS fast data journey, we introduced Amazon ElastiCache for Redis, a fully managed, in-memory data store that operates at microsecond latency. Since then, we have added support for Redis clusters, enabling customers to run faster and more scalable workloads. ElastiCache for Redis cluster configuration supports up to 15 shards and enables customers to run Redis workloads with up to 6.1 TB of in-memory capacity in a single cluster. While Redis cluster configuration enabled larger deployments with high performance, resizing the cluster required backup and restore, which meant taking the cluster offline. Earlier this month, we announced online cluster resizing within ElastiCache. ElastiCache for Redis now provides the ability to add and remove shards from a running cluster. You can now dynamically scale out and even scale in your Redis cluster workloads to adapt to changes in demand. ElastiCache resizes the cluster by adding or removing shards and redistributing keys uniformly across the new shard configuration, all while the cluster continues to stay online and serve requests. No application changes are needed. Scaling with elasticity Having closely watched ElastiCache evolve over the years, I am delighted to see ElastiCache being used by thousands of customers – including the likes of Airbnb, Hulu, McDonalds, Adobe, Expedia, Hudl, Grab, Duolingo, PBS, HERE, and Ubisoft. ElastiCache for Redis delivers predictable microsecond latencies and is super easy to use. Our customers are using ElastiCache for Redis in their most demanding applications, supporting millions of users. Whether it is gaming, adtech, travel, or retail—speed wins, it's simple. As the use cases for Redis continue to grow, customers have demanded more flexibility in scaling their workloads dynamically, while continuing to be highly available and serving incoming traffic. To give you some examples, I've been talking to a few gaming companies lately, and their conversations are about the need for speed and flexibility in scaling, both in and out. They deal with high variability in workloads based on game adoption or seasonality, such as upcoming holidays. If a game leaderboard surges because of a new game title, and tons of players flock to play the game, gaming platforms want to resize the cluster online to handle the bigger load. But as demand decreases, they should just as easily be able to scale-in the environment to optimize costs, all while staying online and serving incoming requests. Our retail customers have shared similar challenges about managing workload surges and declines driven by big sale events. Some customers have also shared their experiences of trying to self-manage Redis workloads and implement online cluster resizing, for workloads where offline cluster resizing was not an option. While open source Redis comes with primitives to help reshard a cluster, they are inadequate. In addition to the cost of self-management, customers have to deal with failures during cluster resizing. Failures can leave the cluster in an irrecoverable state, potentially causing data loss and extended downtime until the cluster can be fixed manually. At Amazon, we have always focused on innovating on behalf of the cus[...]



A Decade of Dynamo: Powering the next wave of high-performance, internet-scale applications

Mon, 02 Oct 2017 11:00:00 PDT

Today marks the 10 year anniversary of Amazon's Dynamo whitepaper, a milestone that made me reflect on how much innovation has occurred in the area of databases over the last decade and a good reminder on why taking a customer obsessed approach to solving hard problems can have lasting impact beyond your original expectations. It all started in 2004 when Amazon was running Oracle's enterprise edition with clustering and replication. We had an advanced team of database administrators and access to top experts within Oracle. We were pushing the limits of what was a leading commercial database at the time and were unable to sustain the availability, scalability and performance needs that our growing Amazon business demanded. Our straining database infrastructure on Oracle led us to evaluate if we could develop a purpose-built database that would support our business needs for the long term. We prioritized focusing on requirements that would support high-scale, mission-critical services like Amazon's shopping cart, and questioned assumptions traditionally held by relational databases such as the requirement for strong consistency. Our goal was to build a database that would have the unbounded scalability, consistent performance and the high availability to support the needs of our rapidly growing business. A deep dive on how we were using our existing databases revealed that they were frequently not used for their relational capabilities. About 70 percent of operations were of the key-value kind, where only a primary key was used and a single row would be returned. About 20 percent would return a set of rows, but still operate on only a single table. With these requirements in mind, and a willingness to question the status quo, a small group of distributed systems experts came together and designed a horizontally scalable distributed database that would scale out for both reads and writes to meet the long-term needs of our business. This was the genesis of the Amazon Dynamo database. The success of our early results with the Dynamo database encouraged us to write Amazon's Dynamo whitepaper and share it at the 2007 ACM Symposium on Operating Systems Principles (SOSP conference), so that others in the industry could benefit. The Dynamo paper was well-received and served as a catalyst to create the category of distributed database technologies commonly known today as "NoSQL." Of course, no technology change happens in isolation, and at the same time NoSQL was evolving, so was cloud computing. As we began growing the AWS business, we realized that external customers might find our Dynamo database just as useful as we found it within Amazon.com. So, we set out to build a fully hosted AWS database service based upon the original Dynamo design. The requirements for a fully hosted cloud database service needed to be at an even higher bar than what we had set for our Amazon internal system. The cloud-hosted version would need to be: Scalable – The service would need to support hundreds of thousands, or even millions of AWS customers, each supporting their own internet-scale applications. Secure – The service would have to store critical data for external AWS customers which would require an even higher bar for access control and security. Durable and Highly-Available – The service would have to be extremely resilient to failure so that all AWS customers could trust it for their mission-critical workloads as well. Performant – The service would need to be able to maintain consistent performance in the face of diverse customer workloads. Manageable – The service would need to be easy to manage and operate. This was perhaps the most important requirement if we wanted a broad set of users to adopt the service. With these goals in mind, In January,[...]



As-Salaam-Alaikum: The cloud arrives in the Middle East!

Mon, 25 Sep 2017 10:00:00 PDT

Today, I am excited to announce plans for Amazon Web Services (AWS) to bring an infrastructure Region to the Middle East! This move is another milestone in our global expansion and mission to bring flexible, scalable, and secure cloud computing infrastructure to organizations around the world. Based in Bahrain, this will be the first Region for AWS in the Middle East. The Region will be in the heart of Gulf Cooperation Council (GCC) countries, and we're aiming to have it ready by early 2019. This Region will consist of three Availability Zones at launch, and it will provide even lower latency to users across the Middle East. This news marks the 22nd AWS Region we have announced globally. We already have 44 Availability Zones across 16 geographic Regions that customers can use today. We still have another five AWS Regions (and 14 Availability Zones) in China, France, Hong Kong, and Sweden. Plus another AWS GovCloud (US) Region in the United States is coming online by the end of 2018. I'm also excited to announce today that we are launching an AWS Edge Network Location in the United Arab Emirates (UAE) in the first quarter of 2018. This will bring Amazon CloudFront, Amazon Route 53, AWS Shield, and AWS WAF to the region and add to the 84 points of presence AWS has around the world. Despite this rapid growth, we don't plan to slow down or stop there: we will bring infrastructure everywhere needed to meet our customers' expectations. 2017 continues a busy year for AWS in the Middle East. Back in January we opened offices in the region to serve our rapidly growing customer base. We now have a presence in Dubai, UAE and Manama, Bahrain with teams of account managers, solutions architects, partner managers, professional services consultants, support staff, and various other functions, so that customers can directly engage with AWS. For the new AWS infrastructure Region we will also be hiring datacenter engineers, support engineers, engineering operations managers, security specialists, and many more. We are continually hiring in the Middle East, so those people looking to join our dynamic and rapidly growing team should visit www.amazon.jobs. In addition to infrastructure, offices, and jobs another investment AWS is making for its customers in the Middle East, and around the world is to run our business in the most environmentally friendly way. One of the important criteria in launching this AWS Region is the opportunity to power it with renewable energy. We chose Bahrain in part due to the country's focus on executing renewable energy goals and its readiness to construct a new solar power facility to meet our power needs. I'm pleased to announce that the Bahrain Energy and Water Authority (EWA) will construct a solar farm that will supply renewable energy to power this infrastructure Region. EWA expects to bring the 100 MW solar farm online in 2019, making it the country's first utility-scale renewable energy project. You might not know that AWS has a long history of working with customers in the Middle East. We have been supporting the growth of organizations in this part of the world since the early days of our business. We have supported the development of technology skills across the region with Training and Certification programs to help customers develop skills to design, deploy, and operate their infrastructure and applications on the AWS Cloud. We run a range of programs to give people cloud skills, from AWSome Days – a one-day workshop-based training for technical professionals - to online resources such as webinars, whitepapers, articles, and tutorials that help to educate people about AWS. In the education sector we have been supporting the development of technology and cloud skills amongst tertiary institutes[...]



AI for everyone - How companies can benefit from the advance of machine learning

Mon, 04 Sep 2017 10:00:00 PDT

This article titled "Wie Unternehmen vom Vormarsch des maschinellen Lernens profitieren können" appeared in German last week in the "Digitaliserung" column of Wirtschaftwoche. When a technology has its breakthrough, can often only be determined in hindsight. In the case of artificial intelligence (AI) and machine learning (ML), this is different. ML is that part of AI that describes rules and recognizes patterns from large amounts of data in order to predict future data. Both concepts are virtually omnipresent and at the top of most buzzword rankings. Personally, I think – and this is clearly linked to the rise of AI and ML – that there has never been a better time than today to develop smart applications and use them. Why? Because three things are coming together. First: Users across the globe are capturing data digitally, whether this is in the physical world through sensors or GPS, or online through click stream data. As a result, there is a critical mass of data available. Secondly, there is enough affordable computing capacity in the cloud for companies and organizations, no matter what their size, to use intelligent applications. And thirdly, an "algorithmic revolution" has taken place, meaning it is now possible to train trillions of algorithms simultaneously, making the whole machine learning process much faster. This has allowed for more research, which has resulted in reaching the "critical mass" in knowledge that is needed to kick off an exponential growth in the development of new algorithms and architectures. We may have come a relatively long way with AI, but the progress came quietly. After all, during the last 50 years, AI and ML were fields that had only been accessible to an exclusive circle of researchers and scientists. That is now changing, as packages of AI and ML services, frameworks and tools are today available to all sorts of companies and organizations, including those that don't have dedicated research groups in this field. The management consultants at McKinsey expect that the global market for AI-based services, software and hardware will grow annually by 15-25% and reach a volume of around USD 130 billion in 2025. A number of start-ups are using AI algorithms for all things imaginable – searching for tumors in medical images, helping people learn foreign languages, or automating claims handling at insurance companies. At the same time, entirely new categories of applications are being created whereby a natural conversation between man and machine is taking center-stage. Progress through machine learning Is the hype surrounding AI and ML even justified? Definitely, because they offer business and society fascinating possibilities. With the help of digitization and high-performance computers, we are able to replicate human intelligence in some areas, such as computer vision, and even surpass the intelligence of humans. We are creating very diverse algorithms for a wide range of application areas and turning these individual pieces into services so that ML is available for everyone. Packaged into applications and business models, ML can make our life more pleasant or safer. Take autonomous driving: 90% of car accidents in the US can be traced to "human failure". The assumption is that the number of accidents will decline over the long term if vehicles drive autonomously. In aviation, this has already been reality for a long time. MIT pioneers Erik Brynjolfsson and Andrew McAfee predict that the macroeconomic effect of the so-called "second machine age" will be comparable to what the steam engine once unleashed when it replaced humans' muscular strength ("the first machine age"). Many are uncomfortable with the idea that an a[...]



Improving Customer Service with Amazon Connect and Amazon Lex

Fri, 30 Jun 2017 11:00:00 PDT

Customer service is central to the overall customer experience that all consumers are familiar with when communicating with companies. That experience is often tested when we need to ask for help or have a question to be answered. Unfortunately, we've become accustomed to providing the same information multiple times, waiting on hold, and generally spending a lot more time than we expected to resolve our issue when we call customer service. When you call for customer assistance, you often need to wait for an agent to become available after navigating a set of menus. This means that you're going to wait on hold regardless of whether your issue is simple or complex. Once connected, the systems that power call centers generally don't do a good job of using and sharing available information. Therefore, you often start out anonymous and can't be recognized until you've gone through a scripted set of questions. If your issue is complex, you may end up repeating the same information to each person you talk to, because context is not provided with the handoff. It's easy to end up frustrated by the experience, even if your issue is successfully resolved. At Amazon, customer obsession is a fundamental principle of how we operate, and it drives the investments we make. Making sure that customers have a great experience when they need to call us is something that we've invested a lot of time in. So much so, that in March 2017, we announced Amazon Connect, which is the result of nearly ten years of work to build cloud-based contact centers at scale to power customer service for more than 50 Amazon teams and subsidiaries, including Amazon.com, Zappos, and Audible. The service allows any business to deliver better over-the-phone customer service at lower cost. When we set out to build Amazon Connect, we thought deeply about how artificial intelligence could be applied to improve the customer experience. AI has incredible potential in this area. Today, AWS customers are using the cloud to better serve their customers in many different ways. For instance, Zillow trains and retrains 7.5 million models every day to provide highly specific home value estimates to better inform buyers and sellers. KRY is helping doctors virtually visit patients and accurately diagnose aliments by applying machine learning to symptoms. Netflix is using machine learning to provide highly personalized recommendations to over 100 million subscribers. There are really exciting projects everywhere you look, including call centers. When Amazon Connect launched, we spoke about the integration with Amazon Lex. One of the really interesting trends in machine learning lately has been the rise of chatbots, because they are well suited to fulfilling customer requests with natural language. Amazon Lex, which uses the same conversational technology as Amazon Alexa, is Amazon Web Services' deep-learning powered chatbot platform. By linking Amazon Lex chatbots into the Amazon Connect contact flow, customers are able to get help immediately without relying on menus or specific voice commands. For example, an Amazon Lex driven conversation with your dentist's office might look like this… Connect: "Hello, thanks for calling. Is this Jeff?" Jeff: "Yes" Connect: "I see you have a cleaning appointment this Friday. Are you calling to confirm?" Jeff: "No, actually." Connect: "Ok, what are you calling about?" Jeff: "I'd like to change my appointment to be next Monday." Connect: "No problem, I have availability on Monday July 3rd at 11:00 AM. Does that work? Jeff: "That's fine." Connect: "Great. I have booked an appointment [...]



Stop waiting for perfection and learn from your mistakes

Wed, 28 Jun 2017 10:00:00 PDT

This article titled "Wartet nicht auf Perfektion – lernt aus euren Fehlern!" appeared in German last week in the "Digitaliserung" column of Wirtschaftwoche. "Man errs as long as he doth strive." Goethe, the German prince of poets, knew that already more than 200 years ago. His words still ring true today, but with a crucial difference: Striving alone is not enough. You have to strive faster than the rest. And while there's nothing wrong with striving for perfection, in today's digital world you can no longer wait until your products are near perfection before offering them to your customers. If so, you will fall behind in your market. So if you can't wait for perfection, what should you do instead? I believe the answer is to experiment aggressively with your product development, accepting the possibility that some of your experiments will fail. Anyone who has listened to, or worked with, management gurus know their mantra: Failure is a necessary part of progress. That's true, but there's often a big gap between the management theory and the reality on the ground. People want to experiment and learn from things that go wrong. But in the flurry of day-to-day business, they're not given enough time to really reflect on the cause of an error and what to do differently next time. The solution is to find a systematic approach that prevents errors from repeating themselves. From perfection to anti-fragility In finding such a systematic way, you first need to distinguish between two types of errors that can happen in your company: those of technology and those of human decision-making. The nice thing is: if you know how to deal effectively with the first, you might end up being better in the second, making better decisions. The financial mathematician and essayist Nassim Taleb offers an interesting take on this issue. He has argued that errors are incredibly valuable because they lead to innovation. He uses the term 'anti-fragility' to make his point. Today's digital business models require smaller, frequent releases to reduce risk. That means the technologies underpinning these new business models must be more than just robust. They must be 'anti-fragile'. The main feature of anti-fragile technology is that it can 'err' without falling apart. In fact, a crisis can make it even better. At Amazon, we also require our systems and customer solutions to be anti-fragile, and we do that by designing our systems to stand the test of time. Our systems must be able to evolve and become more resilient to failure. They must become more powerful and more feature-rich over time as a result of learning from customer feedback and any failure modes they may encounter while operating the systems. An example of a German company that has become 'anti-fragile' is HARTING, the world's leading provider of heavy pluggable connectors for machines and plants. HARTING shows how to think a step ahead about the meaning of quality standards in the digital world. Quality and trust are the most important values for this traditional company, and Industry 4.0 and the digital transformation have already been important focus areas for them since 2011. Even though it was hard to accept at first, HARTING has meanwhile realized that errors are inevitable. For that reason, its development switched to agile methods. It also uses the "minimum-viable-product" approach and relies on microservices for its software. Working this way, HARTING can discard things and create new things more easily. All in all, HARTING has become faster. That can be seen with HARTING MICA, an edge computing solution that enables older machines and pla[...]



Amazon DynamoDB Accelerator (DAX): Speed Up DynamoDB Response Times from Milliseconds to Microseconds without Application Rewrite.

Wed, 21 Jun 2017 11:00:00 PDT

Today, I'm excited to announce the general availability of Amazon DynamoDB Accelerator (DAX), a fully managed, highly available, in-memory cache that can speed up DynamoDB response times from milliseconds to microseconds, even at millions of requests per second. You can add DAX to your existing DynamoDB applications with just a few clicks in the AWS Management Console – no application rewrites required. DynamoDB has come a long way in the 5 years since we announced its availability in January 2012. As we said at the time, DynamoDB was a result of 15 years of learning in the area of large scale non-relational databases and cloud services. Based on this experience and learning, we built DynamoDB to be a fast, highly scalable NoSQL database to meet the needs of Internet-scale applications. DynamoDB was the first service at AWS to use SSD storage. Development of DynamoDB was guided by the core set of distributed systems principles outlined in the Dynamo paper, resulting in an ultra-scalable and highly reliable database system. DynamoDB delivers predictable performance and single digit millisecond latencies for reads and writes to your application, whether you're just getting started and want to perform hundreds of reads or writes per second in dev and test, or you're operating at scale in production performing millions of reads and writes per second. Saving crucial milliseconds Having been closely involved in the design and development of DynamoDB over the years, I find it gratifying to see DynamoDB being used by more than 100,000 customers - including the likes of AirBnB, Amazon, Expedia, Lyft, Redfin, and Supercell. It delivers predictable performance, consistently in the single-digit milliseconds, to users of some of the largest, most popular, iconic applications in use today. I've had a chance to interact with many of these customers on the design of their apps. These interactions allow me to understand their emerging needs, which I take back to our development teams to further iterate on our services. Many of these customers have apps with near real-time requirements for accessing data that need even faster performance than single-digit milliseconds. These are the apps that have motivated us to develop DAX. To give you some examples of my interactions, I've been talking to a few ad-tech companies lately, and their conversations are about how they can save milliseconds of performance. For their applications, they have 20-50 ms to decide whether or not to place a bid for an ad. Every millisecond that is spent querying a database and waiting for a key piece of data is time that they could otherwise use to make better decisions, process more data, or improve calculations to place a more accurate bid. These high-throughput, low-latency requirements need caching, not as a consideration, but as a best practice. Caches reduce latencies to microseconds, increases throughput, and in many cases, help customers save money by reducing the amount of resources they have to overprovision for their databases. Caching is not a new concept, and I have always wondered, why doesn't everyone cache? I think the reasons are many, but most follow a similar trend. Although many developers are aware of the patterns and benefits of adding a cache to an application, it's not easy to implement such functionality correctly. It's also time consuming and costly. When you write an application, you might not need or design for caching on day one. Thus, caching has to be shoehorned into an application that is already operational and experiencing load that would necessitate the added benefits. Adding caching when your app is already experi[...]



Expanding the Cloud – An AWS Region is coming to Hong Kong

Wed, 21 Jun 2017 00:00:00 PDT

Today, I am very excited to announce our plans to open a new AWS Region in Hong Kong! The new region will give Hong Kong-based businesses, government organizations, non-profits, and global companies with customers in Hong Kong, the ability to leverage AWS technologies from data centers in Hong Kong. The new AWS Asia Pacific (Hong Kong) Region will have three Availability Zones and be ready for customers for use in 2018. Over the past decade, we have seen tremendous growth at AWS. As a result, we have opened 43 Availability Zones across 16 AWS Regions worldwide. Last year, we opened new regions in Korea, India, the US, Canada, and the UK. Throughout the next year, we will see another eight zones come online, across three AWS Regions (France, China, and Sweden). However, we do not plan to slow down and we are not stopping there. We are actively working to open new regions in the locations where our customers need them most. In Asia Pacific, we have been constantly expanding our footprint. In 2010, we opened our first AWS Region in Singapore and since then have opened additional regions: Japan, Australia, China, Korea, and India. After the launch of the AWS APAC (Hong Kong) Region, there will be 19 Availability Zones in Asia Pacific for customers to build flexible, scalable, secure, and highly available applications. As well as AWS Regions, we also have 21 AWS Edge Network Locations in Asia Pacific. This enables customers to serve content to their end users with low latency, giving them the best application experience. This continued investment in Asia Pacific has led to strong growth as many customers across the region move to AWS. Organizations in Hong Kong have been increasingly moving their mission-critical applications to AWS. This has led us to steadily increase our investment in Hong Kong to serve our growing base of enterprise, public sector, and startup customers. In 2008, AWS opened a point of presence (PoP) in Hong Kong to enable customers to serve content to their end users with low latency. Since then, AWS has added two more PoPs in Hong Kong, the latest in 2016. In 2013, AWS opened an office in Hong Kong. Today we have local teams in Hong Kong to help customers of all sizes as they move to AWS, including account managers, solutions architects, business developers, partner managers, professional services consultants, technology evangelists, start-up community developers, and more. Some of the most successful startups in the world—including 8 Securities, 9GAG, and GoAnimate—are already using AWS to deliver highly reliable, scalable, and secure applications to customers. 9GAG is a Hong Kong-based company responsible for 9gag.com, one of the top traffic websites in the world. It's an entertainment website where users can post content or "memes" that they find amusing and share them across social media networks. 9GAG generates millions of Facebook shares and likes per month, attracts over 78 million global unique visitors, and receives more than 1 billion page views per month. 9GAG has a small team of nine people, including three engineers to support the business, and uses AWS to service their global visitors. GoAnimate is a Hong Kong-based company that allows companies and individuals to tell great visual stories via its online animation platform. GoAnimate uses many AWS services, including Amazon Polly, to allow users to make their visual animations speak. They chose to use AWS in order to focus on developing their platform, instead of managing infrastructure. They believe that they have reduced development time from 20 to 30 percent by having done so. Some of the largest, [...]



Unlocking the Value of Device Data with AWS Greengrass.

Wed, 07 Jun 2017 09:00:00 PDT

Unlocking the value of data is a primary goal that AWS helps our customers to pursue. In recent years, an explosion of intelligent devices have created oceans of new data across many industries. We have seen that such devices can benefit greatly from the elastic resources of the cloud. This is because data gets more valuable when it can be processed together with other data. At the same time, it can be valuable to process some data right at the source where it is generated. Some applications – medical equipment, industrial machinery, and building automation are just a few – can't rely exclusively on the cloud for control, and require some form of local storage and execution. Such applications are often mission-critical: safeties must operate reliably, even if connectivity drops. Some applications may also rely on timely decisions: when maneuvering heavy machinery, an absolute minimum of latency is critical. Some use cases have privacy or regulatory constraints: medical data might need to be stored on site at a hospital for years even if also stored in the cloud. When you can't address scenarios such as these, the value of data you don't process is lost. As it turns out, there are three broad reasons that local data processing is important, in addition to cloud-based processing. At AWS we refer to these broad reasons as "laws" because we expect them to hold even as technology improves: Law of Physics. Customers want to build applications that make the most interactive and critical decisions locally, such as safety-critical control. This is determined by basic laws of physics: it takes time to send data to the cloud, and networks don't have 100% availability. Customers in physically remote environments, such as mining and agriculture, are more affected by these issues. Law of Economics. In many industries, data production has grown more quickly than bandwidth, and much of this data is low value. Local aggregation and filtering of data allows customers to send only high-value data to the cloud for storage and analysis. Law of the Land. In some industries, customers have regulatory or compliance requirements to isolate or duplicate data in particular locations. Some governments impose data sovereignty restrictions on where data may be stored and processed. Today, we are announcing the general availability of AWS Greengrass, a new service that helps unlock the value of data from devices that are subject to the three laws described above. AWS Greengrass extends AWS onto your devices, so they can act locally on the data they generate while still taking advantage of the cloud. AWS Greengrass takes advantage of your devices' onboard capabilities, and extends them to the cloud for management, updates, and elastic compute and storage. AWS Greengrass provides the following features: Local execution of AWS Lambda functions written in Python 2.7 and deployed down from the cloud. Local device shadows to maintain state for the stateless functions, including sync and conflict resolution. Local messaging between functions and peripherals on the device that hosts AWS Greengrass core, and also between the core and other local devices that use the AWS IoT Device SDK. Security of communication between the AWS Greengrass group and the cloud. AWS Greengrass uses the same certificate-based mutual authentication that AWS IoT uses. Local communication within an AWS Greengrass group is also secured by using a unique private CA for every group. Before AWS Greengrass, device builders often had to choose between the low latency of loca[...]



Weekend Reading: Amazon Aurora: Design Considerations for High Throughput Cloud-Native Relational Databases.

Fri, 19 May 2017 15:00:00 PDT

In many high-throughput OLTP style applications, the database plays a crucial role in achieving scale, reliability, high-performance, and cost efficiency. For a long time, these requirements were almost exclusively served by commercial, proprietary databases. Soon after the launch of the AWS Relational Database Service (RDS) customers gave us feedback that they would love to migrate to RDS. Yet, what they desired more, was a reality that unshackled them from the high-cost, punitive licensing schemes, which came with proprietary databases. They would love to migrate to an open-source style database like MySQL or PostgreSQL, if such a database could meet the enterprise-grade reliability and performance these high-scale applications required. We decided to use our inventive powers to design and build a new database engine that would give database systems such as MySQL and PostgreSQL reliability and performance at scale. Meaning, at a level that could serve even the most demanding OLTP applications. It gave us the opportunity to invent a new database architecture that would address to needs of modern cloud-scale applications, departing from the traditional approaches that had their roots in databases of the nineties. That database engine is now known as "Amazon Aurora" and launched in 2014 for MySQL, and in 2016 for PostgreSQL. Amazon Aurora has become the fastest-growing service in the history of AWS and frequently is the target of migration from on-premise proprietary databases. In a paper published this week at SIGMOD'17, the Amazon Aurora team presents the design considerations for the new database engine and how they addressed them. From the abstract: Amazon Aurora is a relational database service for OLTP workloads offered as part of Amazon Web Services (AWS). In this paper, we describe the architecture of Aurora and the design considerations leading to that architecture. We believe the central constraint in high throughput data processing has moved from compute and storage to the network. Aurora brings a novel architecture to the relational database to address this constraint, most notably by pushing redo processing to a multi-tenant scaleout storage service, purpose-built for Aurora. We describe how doing so not only reduces network traffic, but also allows for fast crash recovery, failovers to replicas without loss of data, and fault-tolerant, self-healing storage. We then describe how Aurora achieves consensus on durable state across numerous storage nodes using an efficient asynchronous scheme, avoiding expensive and chatty recovery protocols. Finally, having operated Aurora as a production service for over 18 months, we share lessons we have learned from our customers on what modern cloud applications expect from their database tier. I hope you will enjoy this weekend's reading, as it contain many gems about modern database design. "Amazon Aurora: Design Considerations for HighThroughput Cloud-Native Relational Databases", Alexandre Verbitski, Anurag Gupta, Debanjan Saha, Murali Brahmadesam, Kamal Gupta, Raman Mittal, Sailesh Krishnamurthy, Sandor Maurice, Tengiz Kharatishvili, Xiaofeng Bao, in SIGMOD '17 Proceedings of the 2017 ACM International Conference on Management of Data, Pages 1041-1052 May 14 – 19, 2017, Chicago, IL, USA. [...]



Faster, higher, stronger: How the digitalization of industry is redefining value creation

Wed, 03 May 2017 10:00:00 PDT

This article titled "Wie die Digitalisierung Wertschöpfung neu definiert" appeared in German last week in the "Größer, höher, weiter (bigger, higher, further)" column of Wirtschaftwoche. Germany's "hidden champions" – family-owned companies, engineering companies, specialists – are unique in the world. They stand for quality, reliability and a high degree of know-how in manufacturing. Hidden champions play a significant role in the German economy; as a result, Germany has become one of the few countries in Western Europe where manufacturing accounts for more than 20% of GDP. By contrast, neighboring countries have seen a continuous decline in their manufacturing base. What's more, digital technologies and business models that are focused on Industry 4.0 (i.e., the term invented in Germany to refer to the digitalization of production) have the potential to reinforce Germany's lead even more. According to estimates by Bitkom, the German IT industry association, and the Fraunhofer Institute of Industrial Engineering IAO, Germany's hidden champions will contribute a substantial portion to the country's economic growth by 2025 and create new jobs. At the same time, many experts believe the fundamental potential of Industry 4.0 has not even been fully leveraged yet. The power of persistence versus the speed of adjustment Most of Germany's hidden champions have earned their reputation through hard work: they have been optimizing their processes over decades. They have invested the time to perfect their processes and develop high-quality products for their customers. This has paid off – and continues to do so. However, digital technologies are now ushering in a paradigm change in value creation. Manufacturing can be fully digitalized to become part of a connected "Internet of Things" (IoT), controlled via the cloud. And control is not the only change: IoT creates many new data streams that, through cloud analytics, provide companies with much deeper insight into their operations and customer engagement. This is forcing Mittelstand companies to break down silos between departments, think beyond their traditional activities, and develop new business models. In fact, almost every industrial company in Germany already has a digitalization project in place. Most of these companies are extracting additional efficiency gains in their production by using digital technologies. Other companies have established start-ups for certain activities, or pilot projects aimed at creating showcases. But many of these initiatives never get beyond that point. The core business, which is doing well, remains untouched by all this. And one of the main reasons why is because the people with the necessary IT expertise in Mittelstand companies sometimes are not sitting at the strategy table as often as they should. Will these initiatives be enough to secure the pole position for Germany's Mittelstand? Probably not. Companies in growth markets are catching up. China's industry, for example, is making huge progress – something that took years to achieve in other places. The role of Chinese manufacturers in the worldwide market is changing: from low-cost workbench to global provider of advanced technology. Market leaders from Germany therefore realize they cannot afford to rest on their laurels. Competitors from the software side are also reshuffling the balance of power, because their offerings will create a completely new market alongside the traditional business of Mittelstand toolmakers and mechanical[...]



Coming to STATION F: The first Mentor's Office powered by AWS!

Wed, 12 Apr 2017 01:00:00 PDT

I am excited to announce that AWS is opening its first Mentor's Office at STATION F in Paris! The Mentor's Office is a workplace exclusively dedicated to meetings between AWS experts and the startups. STATION F is the world's biggest startup campus. With this special offer starting at the end of June, at the campus opening, AWS increases the support already available to startup customers in France. All year long, AWS experts will deliver technical and business assistance to startups based on campus. AWS Solutions Architects will meet startup members for face-to-face sessions, to share guidance on how cloud services can be used for their specific use cases, workloads, or applications. Startup members will also have the possibility to meet with AWS business experts such as account managers, business developers, and consultants. They can explore the possibilities of the AWS Cloud and take advantage of our IT experience and business expertise. With these 1:1 meetings, AWS delivers mentoring to startups to help them bring their ideas to life and accelerate their business using the cloud. AWS will also provide startups with all of the benefits of the AWS Activate program, including AWS credits, training, technical support, and a special startup community forum to help them successfully build their business. For more details about the Mentor's Office at STATION F, feel free to contact the AWS STATION F team. With this opening, Amazon continues to build out global programs to support startup growth and to speed up innovation. Startups can also apply to other Amazon programs to boost their businesses, such as: Amazon Launchpad, which makes it easy for startups to launch, market, and distribute their products to hundreds of millions of Amazon customers across the globe. Alexa Fund, which provides up to $100 million in venture capital funding to fuel voice technology innovation. After the launch of AWS in 2006, we saw an acceleration of French startups adopting the cloud. Successful French startups already using AWS to grow their businesses, across Europe and around the world, include Captain Dash, Dashlane, Botify, Sketchfab,Predicsis, Yomoni, BidMotion, Teads, FrontApp, Iconosquare, and many others. They all get benefits from AWS's highly flexible, scalable, and secure global platform. AWS eliminates the undifferentiated heavy lifting of managing underlying infrastructure and provides elastic, pay-as-you-go IT resources. We have also seen start-ups in France using AWS to grow and become household names in their market segment, such as Aldebaran Robotics (SoftBank Robotics Europe). This startup uses AWS to develop new technologies. They are able to concentrate their engineering resources on innovation, rather than maintaining technology infrastructure, which is leading to the development of autonomous and programmable humanoid robots. Cloud is also an opportunity for startups to reach security standards that were not accessible before. For example, PayPlug is an online payment by credit card solution enabling e-merchants to enrich the customer experience by reinventing the payment experience. Such a service requires suppliers to get PCI DSS certification for the "Service Provider" level, a very demanding certification level. Using AWS's PCI DSS Level 1 compliant infrastructure, Payplug has been certified by L'ACPR (L'Autorité de contrôle prudentiel et de resolution, the French supervisory for prudential and resolution authority) as a financial insti[...]