Microservices are the Future! (...and always will be) - Josh Holtzman, PayPal / Xoom

1
Microservices are the Future!

• and always will be

• Josh Holtzman, Director, Infrastructure
  Engineering
• Xoom, a PayPal Service
• 31 January 2017

2
Xoom.com

• A little history

• Digital remittance
• Founded in 2001
• Acquired bluekite.com in 2014
• Acquired by PayPal in 2016

3
Xoom.com

• A little history (translated)

• Digital remittance ? Highly regulated environment
• Founded in 2001
• Acquired bluekite.com in 2014
• Acquired by PayPal in 2016

4
Xoom.com

• A little history (translated)

• Digital remittance ? Highly regulated environment
• Founded in 2001 ? 16 years of code and data
• Acquired bluekite.com in 2014
• Acquired by PayPal in 2016

5
Xoom.com

• A little history (translated)

• Digital remittance ? Highly regulated environment
• Founded in 2001 ? 16 years of code and data
• Acquired bluekite.com in 2014 ? Polyglot code and
  persistence
• Acquired by PayPal in 2016

6
Xoom.com

• A little history (translated)

• Digital remittance ? Highly regulated environment
• Founded in 2001 ? 16 years of code and data
• Acquired bluekite.com in 2014 ? Polyglot code and
  persistence
• Acquired by PayPal in 2016 ? New rules

7
Throwing down the gauntlet

• Break up the monolith(s)

• Decouple teams
• Reduce time to build and deploy
• Understand our resource needs
• Scale appropriately

8
Microservices to the rescue

• Challenges and risks

• Programming paradigms and idioms
• Service discovery
• Monitoring
• Performance
• Infrastructure as code
• Build and deployment pipeline
• Data ownership

9
Programming paradigms and idioms

• Network operations
• Circuit breakers
• Aggressive timeouts
• Retries
• Throttles
• API designs
• RPC vs REST
• Batch operations
• Response code granularity
• Contracts
• Packaging
• Metadata
• Management uniform

10
Service discovery

• The service-proxy solution

• Custom, local, layer seven load balancers
• Zookeeper back-end
• Apache Curator
• Registration, health checks, and routing
• Service Portal
• Integrating with linkerd.io

11
Monitoring

• Grafana and InfluxDB

• Define required measurements
• persistence operations
• remote calls
• service endpoints
• 3rd party service endpoints
• Define metric types
• gauges
• counters
• histograms
• Standard naming scheme
• Self-service dashboards
• Time series explosion

12
Performance

• Throughput and response latency

• Additional network latency has been offset by
• Reduced contention on datastores
• Limiting the scope of database transactions
• Optimization through observability
• Throughput has improved dramatically
• Latency distribution is wider
• Latency sensitive APIs are deployed nearby

13
Infrastructure as code

• TDD isnt just for applications
• Terraform and Packer for host provisioning on AWS
  and Vsphere
• Puppet and Ansible acceptance testing using
  beaker
• Network gear
• Standardize app packaging
• Docker
• Contracts for deployment
• Application control plane

14
Build and deploy pipeline

• Git-flow
• Branch per feature
• Docker-flow
• Container per branch
• Seed jobs
• Build job per branch
• Automated and self service deployments
• Dev and QA teams can choose branches to deploy
  and test
• Fidelity of environments
• Environment fidelity ? automation success

15
Build and deploy pipeline
16
Data ownership

• Hard problem
• Start eliminating cross-domain joins now
• Two years on, we are just now migrating the last
  auth-server client from tables to APIs
• Analytics becomes more complicated

17
Current status

• 100 distinct microservices across 3 production
  data centers
• Most new features are developed as microservices
• Monoliths still exist, but are being chipped away

18
Lessons learned

• Measure everything, and be prepared to scale your
  monitoring system
• Application packaging contracts and delivery
  pipelines are mandatory
• Staff a tooling team for build, test, and
  deployment automation
• Enroll your network operations team
• The infrastructure and culture we built in order
  to move to microservices has paid off
• Elimination of the monoliths isnt that important

19
Questions
20

• Josh Holtzman
• jholtzman_at_paypal.com