Amazon%20Web%20Services:%20Building%20Highly%20Scalable%20Web%20Applications%20Institutional%20Web%20Management%20Workshop%20July%202007

1
Amazon Web ServicesBuilding Highly Scalable Web
ApplicationsInstitutional Web Management
WorkshopJuly 2007

• Jeff Barr
• Senior Web Services Evangelist
• jbarr_at_amazon.com

2
Today

• Introduction
• Amazon Web Services overview
• Amazon EC2
• Amazon S3
• Amazon SQS
• Application Architecture
• Questions

3
Who am I?

• Software development background
• 5 Years with Amazon
• Senior Developer
• Web Services Evangelist
• Senior Web Services Evangelist
• Veteran of several startups
• Microsoft .Net Team
• Visix Software co-founder

4
Amazon?

• Consumers (Buy)

• Associates, Sellers Merchants (Sell)

• Developers (Innovate)

5
Issues Facing Developers

• 70 of Web Development Effort is Muck
• Data Centers
• Bandwidth / Power / Cooling
• Operations
• Staffing
• Scaling is Difficult and Expensive
• Large Up-Front Investment
• Invest Ahead of Demand
• Load is Unpredictable

6
Dream or Nightmare?

• Slashdot/Digg/TechCrunch Effect
• Rapid, unexpected customer demand/growth

7
Seasonal Spikes
8
More Seasonal Spikes

• Every year, we take the busiest minute of the
  busiest hour of the busiest day and build
  capacity on that, we built our systems to (handle
  that load) and we went above and beyond that.
• -- Scott Gulbransen
• Intuit Spokesman

http//news.com.com/2100-1038_3-6177341.html
9
Solution Web-Scale Computing

• Scale capacity on demand
• Turn fixed costs into variable costs
• Always available
• Rock-solid reliability
• Simple APIs and conceptual models
• Cost-effective
• Reduced time to market
• Focus on product core competencies

10
Amazon Web Services Are...

• A set of APIs and business models which give
  developer-level access to Amazons infrastructure
  and content

• Search As A Service
• Alexa Web Information Service
• Alexa Top Sites
• Alexa Site Thumbnail
• Alexa Web Search Platform

• Data As A Service
• Amazon E-Commerce Service
• Amazon Historical Pricing

• Infrastructure As A Service
• Amazon Simple Queue Service
• Amazon Simple Storage Service
• Amazon Elastic Compute Cloud

• People As A Service
• Amazon Mechanical Turk

11
The Commercial Side

• Standard licensing terms
• Commercially usable
• Aggressive pricing
• Monthly credit card billing
• Self-serve model
• Sign up as developer
• Choose services
• Agree to service licenses
• Enter payment info
• Start coding

12
Web Services Billing
13
Infrastructure Services
Elastic Compute Cloud
Compute
Simple Storage Service
Simple Queue Service
Store
Message
14
Amazon Simple Storage ServiceS3
15
Amazon Simple Storage Service
.15 per GB per month storage

• Object-Based Storage
• 1 B 5 GB / object
• Fast, Reliable, Scalable
• Redundant, Dispersed
• 99.99 Availability Goal
• Private or Public
• Per-object URLs ACLs
• BitTorrent Support

.10 - .18 per GB data transfer
16
Amazon S3 Concepts

• Objects
• Opaque data to be stored (1 byte 5 Gigabytes)
• Authentication and access controls
• Buckets
• Object container any number of objects
• 100 buckets per account / buckets are owned
• Keys
• Unique object identifier within bucket
• Up to 1024 bytes long
• Flat object storage model
• Standards-Based Interfaces
• REST and SOAP
• URL-Addressability every object has a URL

17
S3 SOAP/Query API

• Service
• ListAllMyBuckets
• Buckets
• CreateBucket
• DeleteBucket
• ListBucket
• GetBucketAccessControlPolicy
• SetBucketAccessControlPolicy
• GetBucketLoggingStatus
• SetBucketLoggingStatus
• Objects
• PutObject
• PutObjectInline
• GetObject
• GetObjectExtended
• DeleteObject
• GetObjectAccessControlPolicy

18
(No Transcript)
19
(No Transcript)
20
Amazon Simple Queue ServiceSQS
21
Amazon Simple Queue Service
.10 per 1000 messages

• Scalable Queuing
• Elastic Capacity
• Reliable, Simple, Secure
• Inter-process messaging, data buffering,
  architecture component

.10 - .18 per GB data transfer
22
Amazon SQS Concepts

• Queues
• Named message container
• Persistent
• Messages
• Up to 256KB of data per message
• Peek / Lock access model
• Scalable
• Unlimited number of queues per account
• Unlimited number of messages per queue

23
Amazon SQS At Work
24
SQS SOAP/Query API

• Queues
• ListQueues
• DeleteQueue
• SetVisibilityTimeout
• GetVisibilityTimeout
• Messages
• SendMessage
• ReceiveMessage
• DeleteMessage
• PeekMessage
• Security
• AddGrant
• ListGrants
• RemoveGrant

25
Amazon Elastic Compute CloudEC2
26
Amazon Elastic Compute Cloud
.10 per server hour

• Virtual Compute Cloud
• Elastic Capacity
• 1.7 GHz x86
• 1.7 GB RAM
• 160 GB Disk
• 250 MB/Second Network
• Network Security Model
• Time or Traffic-based Scaling, Load testing,
  Simulation and Analysis, Rendering, Software as a
  Service Platform, Hosting

.10 - .18 per GB data transfer
27
Amazon EC2 Concepts

• Amazon Machine Image (AMI)
• Bootable root disk
• Pre-defined or user-built
• Catalog of user-built AMIs
• OS Fedora, Centos, Gentoo, Debian, Ubuntu,
  Windows Server
• App Stack LAMP, mpiBLAST, Hadoop
• Instance
• Running copy of an AMI
• Launch in less than 2 minutes
• Start/stop programmatically
• Network Security Model
• Explicit access control
• Security groups
• Inter-service bandwidth is free

28
Root-level access
29
Amazon EC2 At Work

• Startups
• Cruxy Media transcoding
• GigaVox Media Podcast Management
• Fortune 500 clients
• High-Impact, S hort-Term Projects
• Development Host
• Science / Research
• Hadoop / MapReduce
• mpiBLAST
• Load-Management and Load Balancing Tools
• Pound
• Weogeo
• Rightscale

30
EC2 SOAP/Query API

• Images
• RegisterImage
• DescribeImages
• DeregisterImage
• Instances
• RunInstances
• DescribeInstances
• TerminateInstances
• GetConsoleOutput
• RebootInstances
• Keypairs
• CreateKeyPair
• DescribeKeyPairs
• DeleteKeyPair

• Image Attributes
• ModifyImageAttribute
• DescribeImageAttribute
• ResetImageAttribute
• Security Groups
• CreateSecurityGroup
• DescribeSecurityGroups
• DeleteSecurityGroup
• AuthorizeSecurityGroupIngress
• RevokeSecurityGroupIngress

31
(No Transcript)
32
Sample Web-Scale ArchitectureGigaVox Media
33
Web-Scale Architecture
34
GigaVox Economics

• Implemented Amazon S3, Amazon EC2 and Amazon SQS
  in November 2006
• Created an infinitely scalable infrastructure for
  less than 100 - building the same infrastructure
  themselves would have cost thousands of dollars
• Reduced staffing requirements - far less
  responsibility for 24x7 operations

35

• Q A

36
Thank You!

• Pointers
• Portal
• http//aws.amazon.com
• Blog
• http//aws.typepad.com
• EC2
• http//aws.amazon.com/ec2
• S3
• http//aws.amazon.com/s3
• Resource Center
• http//aws.amazon.com/resources
• Forums
• http//aws.amazon.com/forums

Jeff Barr Senior Web Services Evangelist jbarr_at_ama
zon.com