Behind-the-Scenes at Salesforce.com R&D: Powering 150 180

1
Behind-the-Scenes at Salesforce.com
RD Powering 150 180 Million Transactions a Day

• Claus Moldt, Salesforce.com

2
Safe Harbor Statement
Safe harbor statement under the Private
Securities Litigation Reform Act of 1995 This
presentation may contain forward-looking
statements including but not limited to
statements concerning the potential market for
our existing service offerings and future
offerings. All of our forward looking statements
involve risks, uncertainties and assumptions. If
any such risks or uncertainties materialize or if
any of the assumptions proves incorrect, our
results could differ materially from the results
expressed or implied by the forward-looking
statements we make. The risks and uncertainties
referred to above include - but are not limited
to - risks associated with possible fluctuations
in our operating results and cash flows, rate of
growth and anticipated revenue run rate, errors,
interruptions or delays in our service or our Web
hosting, our new business model, our history of
operating losses, the possibility that we will
not remain profitable, breach of our security
measures, the emerging market in which we
operate, our relatively limited operating
history, our ability to hire, retain and motivate
our employees and manage our growth, competition,
our ability to continue to release and gain
customer acceptance of new and improved versions
of our service, customer and partner acceptance
of the AppExchange, successful customer
deployment and utilization of our services,
unanticipated changes in our effective tax rate,
fluctuations in the number of shares outstanding,
the price of such shares, foreign currency
exchange rates and interest rates. Further
information on these and other factors that could
affect our financial results is included in the
reports on Forms 10-K, 10-Q and 8-K and in other
filings we make with the Securities and Exchange
Commission from time to time. These documents are
available on the SEC Filings section of the
Investor Information section of our website at
www.salesforce.com/investor. Salesforce.com, inc.
assumes no obligation and does not intend to
update these forward-looking statements, except
as required by law.
3
Claus Moldt VP, Technical Operations
4
Data Centers

• Best of Breed Data Centers

5
Fully Mirrored Cloud Computing Infrastructure
Continued Investments. Unparalleled Confidence.

• Unmatched Reliability
• Two (soon to be 3) mirrored production data
  centers plus a production-class lab facility
• Near real time replicationbetween facilities
• Validated disaster recovery
• MPLS based backbone

• Maximum Uptime Performance
• Carrier neutral network strategy
• No single points of failure
• Carrier level scalability
• Extensive use of high availability server and
  network technologies

• Trusted Security
• World-class security specs
• SAS 70 Type II and SysTrust Certified
• ISO 27001 Certified
• Secure point-to-point data replication
• Secure custody of customer data and backups

Production-Class RD Lab Tape Archive (CA)
Back-Up Production Data Center (VA)
Asia Pacific Production Datacenter
(Singapore Winter 08)
Main Production Data Center (CA)
6
Security FacilitiesMaximum Facilities Security

• 24 x 365 on-site security
• All doors, including cages, are secured with
  biometric hand geometry readers.
• Five levels of biometric scanning including
  man-traps required to reach Salesforce cages
• Fully anonymous exteriors
• Digital camera (CCTV) coverage of entire facility
• Entire perimeter bounded by concrete
  bollards/planters
• A silent alarm and automatic notification of
  appropriate law enforcement officials protect all
  exterior entrances.
• CCTV integrated with access control and alarm
  system.
• Motion-detection for lighting and CCTV coverage.

7
World-Class InfrastructureDelivering leading
On-Demand availability

• Two mirrored data centers plus a production-scale
  lab facility
• 18,000 total sq. feet of cage space
• Mirroring is about more than just having a copy
  of your data
• Salesforce maintains a full-scale replica of the
  production facility as well as your data
• Power Diesel Generators for backup power supply
• Next generation UPS systems (N1)
• Five- Hitec Rotary Continuous Power Supplies
  rated for 4,980kW  (n 1)
• Rotating fly-wheel generator provides UPS and
  Diesel generator start-up
• Two- Detroit Diesel engine 2mW Generators for a
  total of 4,980kW (n 1)
• Eliminates potentially risky UPS battery
  maintenance
• 25,000 gallon diesel fuel tanks supported by two
  fuel vendors
• Cooling
• Precision, N1 HVAC
• Guaranteed by backup water supply
• On-site dedicated wells

8
NetworkIndustry leading performance, scalability
and redundancy

• Carrier-class and carrier-neutral model multiple
  transit vendors
• AboveNet
• MCI
• Level 3
• NTT
• Equinix Exchange
• Sprint
• Multi-gigabit IP transit for external customer
  service
• Lightning-fast performance worldwide
• Data centers located at core Internet hubs
• Access to thousands of global Internet peering
  points delivering global high performance access
• Private peering with key carriers and partners
  (15)
• MPLS/VPLS based backbone
• Enables near real-time replication for
  availability and disaster recovery

9
Scalability

• Highly Scalable POD Architecture

10
Cloud Computing Serves Companies of All Sizes
65,000
30,000
Enterprise Std
30,000
ENTERPRISE MARKET
MID-MARKET
5,800
9,000
5,500
6,300
SMALL BUSINESS
4,000
3,000
3,200
3,500
Number of Subscribers
11
We built the platform for the cloud
YOU get to focus on innovation
We do Infrastructure Services
We do Application Services
We do Operations Services
Network Storage Operating System Database App
Server Web Server Data Center
Security Sharing Integration Customization Web
Services API Multi-Language
Authentication Availability Monitoring Patch
Mgmt Upgrades Backup NOC
Build your data model Build your business
logic Build your user interface

Force.com allowed us to create and deliver a
total of 14 applications all without the
expense and hassles of traditional application
development.

12
The Cloud Computing Model Multi-tenant,
Subscriptions
Subscription
Multi-tenant
Faster Vendor Innovation Economies of Scale
Scalability Automatic Upgrades
13
The Fastest, Easiest and Lowest Risk Path to IT
Success
Client/Server App Server Platforms
Platform as a Service
Source 3rd party analyst surveys
Source Salesforce.com Customer Relationship
Survey conducted in Feb. 2008, by an independent
third-party CustomerSat Inc.
14
Cloud Computing Enables Reactive Innovation26
Major Releases in 9 Years
No Customers left behind Every customer on the
latest version of salesforce.com
All Customizations Upgraded Automatically
15
Proven Scalability and PerformanceDelivering
180 Million Transactions Daily
Page Response Time(ms)
Quarterly Transactions (billions)
Fiscal Year
2005
2006
2007
2008
16
Multi-Tenant Integration Proven Success
Over 2.2 Billion API Transactions per Month
API Transactions
3,000,000,000
2,750,000,000
Page Views
2,500,000,000
I think API font should be bold and line
stronger. Make page views gray or something to
highlight that API transactions is the key thing
to focus on.
2,250,000,000
1,100,000
2,000,000,000
Subscribers
1,750,000,000
1,500,000,000
1,250,000,000
1,000,000,000
750,000,000
-
Q2FY06
Q3FY06
Q4FY06
Q1FY07
Q2FY07
Q3FY07
Q4FY07
Q1FY08
17
Enterprise Scalability Performance
Your Company
1
Your Division
2
Query Optimization Engine
Your Data
Your Customizations
3
Massive Scale
Immediate Response
Your Sharing Model
4
Sub-second response time
Billions of Transactions
18
Scalable Software ArchitectureUtilizing Industry
Standard Platforms for High Availability

• Database Server Oracle RAC EE, Dell, Sun
• Clustering SunCluster
• Web Site and Application Server Dell, Resin
• Search Server Jakarta Lucene
• Storage Management Hitachi Data Systems, Sun
• Backup Software Veritas/RMAN
• Operating Systems
• Sun Solaris
• Redhat Linux

19
Pod Architecture further enhances availability,
horizontal scale, and platform for future growth
Network Services
Sandbox Pod
Storage Services
Backup Services
Monitoring Services
Threshold User Capacity Add a POD
Salesforce.com confidential
20
Whats Shared Across Pods

• Ops Stack
• syslog
• bastion
• jump/kick start
• release
• backup
• DNS
• TACACS
• SecurID

• Storage HDS 9990
• SAN Cisco MDS
• Core Network Force10
• Edge Network Juniper
• Search Indexer Sun SPARC
• WWW Services Dell/Linux
• Edge Firewalls Juniper Netscreens
• Load Balancers F5
• Proxy Services Dell/Linux
• IDS
• BlueCoat
• Performance Monitoring
• Email

Salesforce.com confidential
21
Redundancy

• Network Redundancy/Multiple Carriers
• Load Balancing/Fail-Over
• Clustering/RAC EE
• MirrorForce

22
SFDC Built for High Availability

• Multiple Network Carriers
• Redundant Routers at Entry Points
• Fail-over Configured Firewalls
• Redundant Load Balanced Load Balancers
• Redundant Hubs/Switches at VLANs
• Web, Application, API, Search, Cache, Index,
  Batch Servers
• Load Balanced, Fail-over or Clustered
• Data Base Servers
• Oracle RAC EE running on 4 way Clustered Nodes
• Sized to sustain Peak Load if Node failure
• Storage
• Multiple paths for reliability
• 4 inter-connects per DBMS Server
• Alternate paths to separate Storage Directors
• 2 Storage Directors per Array

23
Multiple Network Carriers and Redundancy at the
Edge
24
Sample POD Architecture Built for Redundancy
25
Backup and Disaster Recovery Strategy

• Near real time replication between data centers
• Disaster Recovery Strategy Failover to
  full-scale east coast replica data center backup
  facility.

Near real time replication between data centers
East Coast
Backup DR Data Center
West Coast
Production Data Center
OC48/MPLS/VPLS Backbone
Lab and Tape Archive
Local 48 Hour lag standby databases
San Francisco
26
Monitoring

• Performance Management
• Pro-active Monitoring

27
Performance and MonitoringEnd-to-End Monitoring
Guarantees Uptime and Security

• Monitoring Strategy Multi-Tier Monitoring
  Strategy
• Nagios monitoring software
• Gomez performance software service
• EMC Smarts
• Coradiant End-User experience
• Custom instrumentation within the Application
• Performance Metrics
• Average page load times between 250 and 400
  milliseconds
• 180M Transactions Daily
• Pages served in fiscal Q1 2008 5.4 Billion
• 47,600 Customers
• 1,100,000 Subscribers
• 95 Customer Satisfaction
• Open Communication
• http//trust.salesforce.com

Cricket
Custom Agents
January 2005 independent survey
28
Example of SFDC Monitoring Pro-Agents

29
Trust Site - Incident Communications Example
30
Trust Site - Security Alerts and Examples
31
Capacity Planning
Capacity forecasting
Capacity Planning and Analysis Model
System/App Outputs

• Basic load management data used to forecast
  annual growth.
• Detailed customer transaction level monitoring
  for focused and proactive capacity management
• Granular performance breakdowns by transaction
  type
• Impact analysis of custom transactional logics
• Operationally efficient and scalable

1. Annual demand growth for all enterprise
customers are projected and reviewed every week
2. Three year forecast are predicted for
Datacenter capacity
3. Results analyzed and systems scaled
appropriately to meet demand growth.
Multiple detailed metrics
32
Security

• Overview

33
Security

• Dedicated Security Organization
• Strategy/Charter
• Mitigate risks while complying with legal,
  statutory, contractual, and internally developed
  requirements 
• Develop and enforce policies and procedures
• Design and secure information systems using
• security domains, defense in-depth and least
  privilege principles
• Develop and integrate security architecture into
  business processes (CobiT, ISO27001)
• Conduct employee security awareness training
  classes
• Perform regular vulnerability assessments and
  audits
• Addresses all layers
• Physical Security
• Logical Network Security
• Host Security
• Transmission Level Security
• Database Security

34
Internal Vulnerability Assessments

• Salesforce.com implements a multi-prong approach
  to ensure the software we release is secure. 
  Specifically, we perform the following tasks to
  assure security in the development lifecycle.
• Architecture Reviews Salesforce.com architects
  (including security team) meet regularly to
  discuss features that could be considered high
  risk.
• Development  Salesforce.com developers follow
  coding best practices such as those specified in
  OWASP.  All code prior to check in is reviewed. 
  Code quality and security tools (Findbugs,
  Checkmarx.) are run frequently to detect possible
  program anomalies. All developers receive
  application security training to help them write
  secure code.
• Quality Assurance Salesforce.com QA testers
  analyze their features through both positive and
  negative testing.  Salesforce.com also employs
  several black box analysis tools (Appscan, Peros,
  etc.) to help in identification of security
  vulnerabilities.
• Information Security Salesforce.com InfoSec tests
  medium and high risk features. (Proprietary
  fuzzers, Burp Suite) Periodically brings in
  third parties to perform code reviews, blackbox
  analysis and design reviews (iSEC Partners, etc.)

35
External Vulnerability Assessments

• MSSPs include SPI Dynamics, Solutionary, Symantec
• Network Assessments and Application Assessments
• Assessments cover the following

• Cross-Site Scripting
• Input validation
• Buffer Overflow
• SQL Injection
• Directory Traversal
• Parameter Overflow
• Path Manipulation
• Command Execution
• Path Truncation
• Character Encoding
• Character Stripping
• Site Search
• Application Mapping
• Automatic Form-Filling
• Configuration Management
• Proxy Support
• Parameter Injection

• Directory Enumeration
• Authentication and Session Management
• Web Server Assessment
• HTTP Compliance
• SSL Support and Strength
• Certificate Analysis
• Content Investigation
• Spam Gateway Detection
• Developer Comments
• Absolute Path Detection
• Error Handling
• Permissions Assessment
• Brute Force Authentication attacks
• Known Attacks
• Session Hijacking
• Horizontal Attacks
• Insecure Storage

• Executive Summaries available upon request

36
Managing Change

• Release Management/Change Management
• Maintenance Windows

37
SFDC Release Testing/Managing Quality Change

• SFDC Testing is focused to ensure transparency of
  changes
• Intense Functional and System Testing prior to
  release
• Forward and backward compatibility of all
  standard APIs
• Review Teams
• Metrics and Reporting
• Quality Targets
• System metric/trends
• All production changes logged in cases
• Includes rollback, validation and expected impact

38
Salesforce.com Releases
Example
Frequency
TYPICAL Time of the week
Release objective
Release
Planned 146 release
Approx 3 - 4 per year
Friday night, Saturday
Significant new functionality and enhancement
Major
End user experience enhancement with 146.8 release
Weekly for first 3-4 weeks after major
release Every other week there after
Wednesday evening (No Downtime)
Bug fixes or minor functionality enhancement
Patch/Dot
Break fix errors
Unscheduled on as needed basis
(No Downtime)
Fix production vulnerabilities
E Release
Salesforce.com confidential
39
Maintenance Windows are Designed to Minimize
Business Disruption to Customers

• Established based on analysis of our customer
  usage patterns and traffic
• 4 hour windows reserved for routine maintenance
• 1st 3rd Saturdays
• 7pm to 11pm Pacific Time all NA EU Instances
  except NA2
• 12am to 0400am Sunday Pacific Time NA2 only
• 10am to 2pm Saturday Pacific Time AP0 only
• Plans for EMEA instance can be adjusted to fit
  their time zones (for maintenance of non-shared
  infrastructure)
• Maintenance of Shared Infrastructure 1st 3rd
  Saturdays 7pm to 11pm Pacific Time
• Windows are planned conservatively
• Not all reserved windows are utilized
• Actual maintenance downtime is a fraction of
  declared window
• Future roadmap to minimize and eventually
  eliminate downtime

Note Product release updates (3 per year)
typically occur on a separate schedule on Friday
nights and have longer windows
40
Maintenance Windows are Declared 1 Week in
AdvanceSample Notification
41
Summary
42
System Availability

• What does this mean?
• Highly reliable system that your Business can
  depend on
• Year to Date Planned System Availability
• 99.9

43
Why You Shouldnt Do it Yourself

• Companies and organizations implementing and
  managing enterprise software face three core
  challenges
• Rising COST
• Increasing COMPLEXITY
• Regulatory COMPLIANCE

44
Leave the heavy lifting to us

• Data Center Management
• Provisioning and installation of hardware, O/S,
  and network equipment within SFDC datacenter
• O/S, database, and network monitoring
• Application Management
• Patch Management
• Application Upgrades
• API Management Upgrade validation
• Capacity Planning and tuning
• Security infrastructure and application features
• Configuration and management of storage
• Backup and recovery services
• Disaster Recovery (100 Performance Capacity)
• O/S and Internet Backbone network support
• Technology Refresh
• Skilled Subject Matter Work Force
• 7x24 hour Support at all levels
• Why should you Invest This is
  our Core !

45
Thank You

• QA