Moving Pictures

1
Moving Pictures

• Implementing Video on Flickr
• Cal Henderson

2

• Hello

3
Flickr

• Large scale kitten sharing website
• Started 2003, launched 2004
• 5 years old this december
• Almost 3 billion photos

4
Enter Video

• Video was added this year
• Launched April 2008
• Many hundreds of thousands of videos uploaded
• Many millions of playbacks

5
(No Transcript)
6
(No Transcript)
7
(No Transcript)
8
(No Transcript)
9
(No Transcript)
10

• Video? Thats just like photos!
• -Me, before Flickr Video

11
12 4 Steps

• 4 main tasks
• Uploading
• Transcoding
• Storage
• Serving Playback

12

• 1.
• Upload

13
Simple upload

• Web forms
• Just like any other file
• enctype"multipart/form-data"
• type"submit" value"Go!" /
• But files are large / huge

14
Issues

• Two components for uploading
• Sending from the client
• Receiving on the server
• Both of these present problems

15
Sending from the client

• Multiple options
• Simple form
• Flash
• Desktop app

16
Simple Forms

• Pros
• Very easy to implement
• Works on every browser out of the box
• Cons
• Upload progress is harder
• Slow
• Select a single file at once

17
Flash

• Pros
• Upload progress is easy
• Fast
• Multi select of files
• Cons
• Harder to implement
• Flash isnt quite ubiquitous

18
Desktop App

• Pros
• Upload progress is easy
• Very fast
• Multi select of files
• Drag and drop
• Cons
• Hard to develop
• Hard to deploy (relative to the web)

19
Making Progress

• Upload progress
• Not impossible with plain forms
• Just need to be able to query the upload progress
  via AJAX
• Multiple machines
• The VIP issue
• Enter Perlbal

20
Making Progress
1. Browser starts upload
Web 1
Browser
2. Web server broadcasts progress via UDP
Load balancer
Web 2
3. Browser queries progress via AJX call
21
Receiving on the server

• File uploads are slow
• Much slower than serving pages
• Apache processes are heavy
• Waste of resources
• Use a poll based server (like jetty)

22
Receiving on the server

• Or, use a buffering layer
• Perlbal is great for this
• Or a lightweight Apache
• E.g. w/ mod_proxy

Slow
Fast
Browser
Buffer
Server
23
But wait

• Its not just the first step thats slow
• Moving files around between servers is slow
• Do it out of band
• Asynchronous jobs are in order anyway
• Do it!

24

• 2.
• Transcode

25
Transcode?

• Why transcode at all?
• Input comes from many sources
• Point shoots
• DV Cams
• Mobile devices
• Video editing software
• All in different formats

26
So many formats

• But very few of these formats can be played back
  cross platform
• Without special software or hardware
• Formats are designed to do one thing well
• They dont always manage even that
• Transcoding puts all videos on an equal footing

27
Video file basics

• Most file types are really just containers
• MOV, FLV, AVI
• The data inside can be in multiple formats
• We call these codecs (encoder decoder)
• Files contains multiple streams
• Both audio and video

28
Interleave

• Audio and video are often interleaved
• Hence AVI
• A video file looks like this
• Headers
• Video chunk
• Audio chunk
• Video chunk
• Audio chunk
• Etc

29
Compress

• Raw video files are huge
• A bitmap for every frame
• Rarely used, even in post production
• At 30 fps, that gets crazy pretty quickly
• We dont need to store every frame
• Static backgrounds dont change (much) between
  frames

30
Compresssss

• Intraframe
• Treat each frame as a picture
• Compress it (just like JPEG)
• DCT (Discrete Cosine Transform)
• Interframe
• Store the differences between frames
• Treat the pixels as a 3D array to be compressed

31
The IPB

• Three frame types I, P B
• Intra coded pictures
• A full raw frame
• Predicted pictures
• Based on a single reference frame
• Bi-predictive pictures
• Based on two or more reference frames

32
IPBIPBIPBIPB

• Reference frames may be I, P or B
• P B frames may contain a mix of image data and
  motion vector displacements
• I frames require the most bits
• Then P frames
• Then B frames

33
Bad terminology

• We should really say picture
• (Not frame)
• Because of interlacing
• Really, we encode fields not frames
• Picture is the general term
• And H.264 contains slices
• Sub-regions of the field
• Macroblocks Artifacts

34
I-Frames

• Also called Key Frames
• Allow easy random seeking
• Twice a second for Digital TV DVDs
• More widely spaced online

35
Seekable
36
Seekable
37
Oh, and audio too

• We can worry less about this
• Older problem, well solved
• MP3 is pretty good
• Who cares how it works?
• Syncing is the only issue
• Presentation Time Stamps (PTS) and Decode Time
  Stamps (DTS) in MPEG-2

38
Flash! Woah-oh!

• The big question
• Flash?

39
Non-flash sites

• QuickTime
• Windows Media
• This is gradually disappearing
• Flash player is ubiquitous
• Compression is good enough
• Interactive too
• But no 3D/VR as with QuickTime (

40
The Flash Player

• Flash Player 6
• March 2002
• Video Sorenson Spark (H.263)
• Audio MP3
• Or ADPCM / Uncompressed
• Or Nellymoser Asao

41
Second Generation

• Flash Player 7
• August 2005
• Video On2 TrueMotion VP6
• Audio MP3

42
The hot shit

• Flash Player 9 (update 3)
• December 2007
• Video H.264 (MPEG-4 Part 10)
• w/ container formats from MPEG-4 Part 14
• Audio AAC (MPEG-4 Part 3)
• Plus 3GPP Timed Text (MPEG-4 Part 17)

43
TrueMotion VP6

• Proprietary
• Reasonable compression
• Created by On2
• Patented
• Probably illegal for GPL code
• YouTube uses it for lower quality and old streams

44
H.264

• Not proprietary
• Good compression
• MPEG Standard
• Open, but patented
• Patent licenses from the MPEG LA
• Unclear how this applies to (L)GPL code
• But probably badly
• YouTube using it for higher quality streams
• iPhones and AppleTV

45
Software

• Open source transcode tools
• FFmepg
• libavcodec for VP6
• Probably illegal dubious
• Also pretty shoddy
• Can only decode H.264

46
More software

• MEncoder
• libmpcodecs uses libavcodec
• VLC
• libvlc uses libavcodec
• So basically the same
• Different muxing, same codecs

47
Free H.264?

• Unfortunately, not really
• x264 is the only usable one
• Its pretty good
• MEncoder can use it
• Still limited in options at this point
• Again, dubiously legal

48
Non-free tools

• Flash encoder
• Not automatable
• On2 FlixEngine
• Creators of VP6
• Windows or Linux
• Some support for H.264
• Rhozet Carbon Coder
• The new hot shit
• Good H.264 support
• Windows

49
Choices

• Video codec
• Resolution
• Bitrate (VBR, CBR)
• Keyframes
• Audio codec
• Channels
• Bit depth
• Sampling rate

50
Doing it at scale

• Not really a problem
• Very easily parallelizable
• Amazon EC2 is awesome here
• Exactly what it was design for
• Grow/shrink as needed
• But, per-CPU software licensing

51

• 3.
• Store

52
Easy!

• Really, just like photos
• But with bigger files
• Same disk layout as any other serving
• But the serving part is slower

53
But..

• Remember the files are huge
• Operations take time
• More likely to fail halfway through
• Checksums are your friend
• Do it all asynchronously

54

• 4.
• Serve Playback

55
The choice

• Streaming
• vs
• Progressive

56
Streaming

• Pros
• Easily seekable
• Live feeds
• Cons
• Special server software
• Slower to start
• Firewall troubles

57
Progressive download

• Pros
• Just use a web server
• Play offline
• Firewall/proxy friendly
• Cons
• Harder to seek ahead (but not impossible)

58
Streaming tech

• Non flash stuff
• Well ignore that
• Youre using flash, right?
• RTMP
• Real-time Messaging Protocol
• Proprietary (thanks Adobe!)

59
RTMP

• RTMP - Raw TCP socket stuffs
• RTMPT RTMP tunneled over HHTP
• For firewalls, etc
• Flash Media Server
• previously Flash Communication Server
• Wowza Pro
• 1000/server

60
Open source

• Its not all bad
• Red5
• Java implementation of RTMP server
• Mostly feature complete
• Beta quality, but usable in production
• Facebook

61
Progressive

• Used by the majority of large sites
• Very simple!
• Seekable with server support

62
Seeking

• Serve the FLV starting at a different point
• Just add a simple FLV preamble before seeking
  into the file
• Simple to do in PHP, Perl, etc
• mod_flvx for Apache
• mod_secdownload for lighttpd

63

• 5.
• Other considerations

64
Review

• Videos are slow
• Expensive to review
• Review grids
• Doesnt cover audio

65
(No Transcript)
66
Not enough?

• Social tools are useful here

67

• Summary

68
Summing up

• Flash makes sense
• For uploading too
• H.264 is probably your best bet today
• Transcoding software still costs money
• Unless youre willing to take on the risk
• Progressive download is basically awesome

69
The end!
70
Awesome!

• These slides are available online
• iamcal.com/talks/