Prof. Marc Davis - PowerPoint PPT Presentation

Loading...

PPT – Prof. Marc Davis PowerPoint presentation | free to download - id: 353c4-ZjAxN



Loading


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation
Title:

Prof. Marc Davis

Description:

... language you don't understand and an alphabet you can't read ... 'Audio graffiti' audio tour of the world. Game: Spot the terrorist. Amateur photo contest. ... – PowerPoint PPT presentation

Number of Views:51
Avg rating:3.0/5.0
Slides: 61
Provided by: valuedg233
Category:

less

Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: Prof. Marc Davis


1
Mobile Media Metadata The Future of Mobile
Imaging
  • Prof. Marc Davis
  • University of California at Berkeley
  • School of Information Management and Systems
  • Garage Cinema Research
  • http//garage.sims.berkeley.edu

2
Four Questions
  • How many of you read text every day?
  • How many of you write text every day?
  • How many of you watch video (movies, TV, DVD,
    Internet video, etc.), listen to recorded audio
    (music, speech, etc.), or look at photographs
    everyday?
  • How many of you make video (movies, TV, DVD,
    Internet video, etc.), record audio (music,
    speech, etc.), or take photographs everyday?

3
Garage Cinema Research
  • Research and develop technology and applications
    that will enable daily media consumers to become
    daily media producers
  • Theory, design, and development of digital media
    systems that
  • Create descriptions of media content and
    structure (metadata)
  • Use metadata to automate media production and
    reuse

4
What is the Problem?
  • Today people cannot easily find, edit, share, and
    reuse media
  • Computers dont understand media content
  • Media is opaque and data rich
  • We lack structured representations
  • Without metadata (descriptions of media content
    and structure), manipulating digital media will
    remain like word-processing with bitmaps

5
Signal-to-Symbol Problems
  • Semantic Gap
  • Gap between low-level signal analysis and
    high-level semantic descriptions
  • Vertical off-white rectangular blob on blue
    background does not equal Campanile at UC
    Berkeley

6
Signal-to-Symbol Problems
  • Sensory Gap
  • Gap between how an object appears and what it is
  • Different images of same object can appear
    dissimilar
  • Images of different objects can appear similar

7
Computer Vision and Context
  • You go out drinking with your friends
  • You get drunk
  • Really drunk
  • You get hit over the head and pass out
  • You are flown to a city in a country youve never
    been to with a language you dont understand and
    an alphabet you cant read
  • You wake up face down in a gutter with a terrible
    hangover
  • You have no idea where you are or how you got
    there
  • This is what its like to be most computer vision
    systemsthey have no context
  • Context is what enables us to understand what we
    see

8
Traditional Media Production Chain
METADATA
Metadata-Centric Production Chain
PRE-PRODUCTION
POST-PRODUCTION
PRODUCTION
DISTRIBUTION
9
Research Projects
  • Media Streams
  • A framework for creating metadata throughout the
    media production cycle to enable media reuse
  • Active Capture
  • Automates direction and cinematography using
    real-time audio-video analysis in an interactive
    control loop to create reusable media assets
  • Adaptive Media
  • Uses adaptive media templates and automatic
    editing functions to mass customize and
    personalize media
  • Mobile Media Metadata
  • Leverages the spatio-temporal context and social
    community of media capture to automate metadata
    creation for mobile media
  • Social Uses of Personal Media
  • Analysis of social uses of media to predict
    future uses and shape the design of
    next-generation personal media devices and
    applications

10
Automated Media Production Process
11
The Coming Media Revolution
  • Media capture devices become programmable and
    networked
  • Metadata creation and use become integrated
    throughout media production and reuse
  • Media production changes from being a mechanical
    process to a computational process
  • Media becomes programmable and networked
  • Daily media consumers become daily media producers

12
Research Projects
  • Media Streams
  • A framework for creating metadata throughout the
    media production cycle to enable media reuse
  • Active Capture
  • Automates direction and cinematography using
    real-time audio-video analysis in an interactive
    control loop to create reusable media assets
  • Adaptive Media
  • Uses adaptive media templates and automatic
    editing functions to mass customize and
    personalize media
  • Mobile Media Metadata
  • Leverages the spatio-temporal context and social
    community of media capture to automate metadata
    creation for mobile media
  • Social Uses of Personal Media
  • Analysis of social uses of media to predict
    future uses and shape the design of
    next-generation personal media devices and
    applications

13
Moores Law for Cameras
2000
2002
400
Kodak DX4900
Kodak DC40
40
SiPix StyleCam Blink
Nintendo GameBoy Camera
14
CaptureProcessingInteractionNetwork
15
Camera Phones as Platform
  • Media capture (images, video, audio)
  • Programmable processing using open standard
    operating systems, programming languages, and
    APIs
  • Wireless networking
  • Personal information management functions
  • Rich user interaction modalities
  • Time, location, and user contextual metadata

16
Camera Phones as Platform
  • In the first half of 2003, more camera phones
    were sold worldwide than digital cameras
  • By 2008, the average camera phone is predicted to
    have 5 megapixel resolution
  • Last month Casio and Samsung introduced 3.2
    megapixel camera phones with optical zoom and
    photo flash
  • There are more cell phone users in China than
    people in the United States (300 million)
  • For 90 of the world their computer is their
    cell phone

17
Campanile Inspiration
18
Mobile Media Metadata Idea
  • Leverage the spatio-temporal context and social
    community of media capture in mobile devices
  • Gather all automatically available information at
    the point of capture (time, spatial location,
    phone user, etc.)
  • Use metadata similarity and media analysis
    algorithms to find similar media that has been
    annotated before
  • Take advantage of this previously annotated media
    to make educated guesses about the content of the
    newly captured media
  • Interact in a simple and intuitive way with the
    phone user to confirm and augment system-supplied
    metadata for captured media

19
MMM Demo Video
20
From Context to Content
  • Context
  • When
  • Date and time
  • Where
  • CellID refined to semantic place
  • Who
  • Cellphone user
  • What
  • Activity as product of when, where, and who
  • Content
  • When was the photo taken?
  • Where is the subject of the photo?
  • Who is in the photo?
  • What are the people doing?
  • What objects are in the photo?

21
Space Time Social Space
22
What is Location?
23
Camera Location vs. Subject Location
  • Camera Location Golden Gate Bridge
  • Subject Location Golden Gate Bridge
  • Camera Location Albany Marina
  • Subject Location Golden Gate Bridge

24
Kodak Picture Spot
25
Themed Kodak Picture Spot
26
Location Guesser Concept
  • Calculate weighted sum of features
  • Most recently visited location
  • Most visited location by me in this CellID
    around this time
  • Most visited location by me in this CellID
  • Most visited location by others in this
    CellID around this time
  • Most visited location by others in this CellID

27
Mobile Media Metadata Project (MMM)
  • Develop and test our technology for mobile media
    metadata creation, sharing, and reuse
  • Develop application scenarios that use this
    technology
  • Study the usage patterns of applications for
    mobile media metadata creation, sharing, and
    reuse

28
MMM Fall 2003 Project
  • Numerous application ideas for mobile media
    capture and sharing (at least seven documented in
    detail with persona and scenario descriptions and
    annotated storyboards)
  • A metadata framework for describing and sharing
    mobile media
  • A database of annotated captured media
  • A prototype of one of the student application
    scenarios
  • A write-up of the project including student
    feedback, assessment of application ideas, and
    recommendations for further research and
    development

29
MMM Research Issues
  • Metadata and media creation, sharing, and reuse
  • Ontology design
  • Vocabulary control
  • Leveraging spatial, temporal, and social context
    to infer media content
  • Determining similarity
  • Application ideas and use scenarios
  • User interfaces

30
MMM Research Issues
  • Ontology of photography
  • Who owns/accesses/changes pictures of me?
  • Who owns/accesses/changes my pictures?
  • Who owns/accesses/changes my metadata?
  • Who owns/accesses/changes metadata about me
    and/or my media?
  • Mobile media application design methods
  • HCI methodology for mobile media design
  • Social science methodology for design of future
    applications

31
MMM Initial Application Ideas
  • Metadata creation sharing, and reuse of media and
    metadata from various users connected in space,
    time, and affinity
  • Content-based access to mobile media assets
    enabling
  • Networked photo albums
  • Personalized media messages
  • Personalized media CallerID
  • Matchmaking services
  • Ecommerce
  • Location-based and travel applications for using
    networked media assets and metadata
  • Networked travel albums
  • Networked travel guidebooks

32
Over 150 MMM Application Ideas
  • Real time party ratings with pictures
  • Forgotten historical site cataloguing
  • Mapping wireless access on campus for public
  • Picture/video comparison to map location
  • Tagging people
  • "Audio graffiti" audio tour of the world
  • Game Spot the terrorist
  • Amateur photo contest.
  • Mobile American Idol
  • Affair detection
  • Do I sue or not? Online car estimates
  • Family tree builder/viewer
  • Human Clock Pictures of people showing the time
  • Human Weather What are people wearing in San
    Francisco?
  • Reality show that is following someone around for
    a night going out and party that people could
    watch online
  • Family tree builder/viewer
  • Human Clock Pictures of people showing the time
  • Human Weather What are people wearing in San
    Francisco?
  • Reality show that is following someone around for
    a night going out and party that people could
    watch online
  • Translation. Take a picture of something, and get
    the word for it in the local language. Or enter a
    word in your own language to get a picture which
    you can show to a native
  • Directions/locations. Illustrated travel guides,
    an interactive visual Lonely Planet
  • Indie reporting. News coverage via amateur
    reporters. The blackout provided some interesting
    examples
  • Some "leveling game" like taking photos of people
    wearing gap jeans on the street, if you get
    enough, you win a pair, and go onto the next level

33
MonkeyBotster
  • Monkeybotster aims to allow you to access
    interesting events that have occurred in
    someone's past as well as contribute things from
    the present through two degrees of social
    separation (your friend's friends can see your
    events but not your friend's friend's friends,
    etc.)

34
LunchMeister
  • LunchMeister enables users to share information,
    opinions, and photos of local restaurants in real
    time, allowing them to maximize the use of an
    often limited lunch hour

35
DARE
  • DARE will leverage the camera phones as tools
    that allow users to play games with and against
    each other
  • The main objective of DARE is to bring different
    social networks together informally, or as ice
    breakers and team builders for corporations,
    school groups and other such groups

36
Recipe Box
  • Recipe Box will facilitate cooking for people who
    wish to spend less money and time making
    presentable meals
  • Users will be able to select from various recipes
    in an annotated media database on their mobile
    phones
  • Users can add recipes illustrated with annotated
    media to the database from their mobile phones

37
HouseBuddies
  • HouseBuddies transforms the mobile phone into a
    collaborative apartment-hunting device that
    enables real-time information sharing
  • It facilitates a group of friends in their search
    for a place to live by allowing them to assemble
    a list of house ads from various sources and
    collaborate on investigating their housing options

38
Recreation Evaluation Interface
  • Recreation Evaluation Interface enables camera
    phone users to see what's happening where they
    are not, and let others in their community know
    what's happening where they are
  • Using real-time data and a reference database,
    this community can connect with other people with
    similar interests to maximize their local
    knowledge, enabling better decision-making about
    how and where to spend one's time

39
Wishter
  • Wishter would manage
  • Lists of potential gifts for friends and family
  • A wishlist of items for oneself
  • It would allow the user to annotate a potential
    gift with a physical description, the event for
    which the gift is being considered, and the
    location where the item was seen

40
Wishter Rapid Development
41
MMM User Studies
  • Deployed MMM prototype with 40 graduate students
    and 15 researchers for 4 months
  • Weekly surveys administered to MMM users
  • Current usage patterns gathered
  • 5 subjects for usability testing with MMM users
  • User interaction and user motivation addressed
  • 8 subjects for first focus group of MMM users
  • Current annotation habits discussed
  • User motivation addressed
  • 7 subjects for second focus group
  • Current use of photos (capture, annotation,
    sharing)

42
User Interaction Findings
  • Network unpredictability
  • Slow and intermittent connectivity
  • Frequently dropped service with little feedback
  • User work to that point was often lost
  • Proposed Solution Limit continual network
    interaction by creating a full-client application
    that can use network in background

43
User Interaction Findings
  • Mobile UI usability
  • Hardware buttons were not remapped to support MMM
    UI functions
  • Desktop-based prototyping tools and methods dont
    predict usability issues with mobile devices
  • Proposed Solution Use a prototyping methodology
    that better simulates the specifics of mobile
    interaction to design user experiences and test
    usability

44
User Interaction Findings
  • Presentation of faceted hierarchical structure
  • Lists of 12-15 items should not be exceeded as
    selection devices
  • Proposed Solution Create smaller application
    specific hierarchies to present to the user,
    (which correlate into a broader general structure
    on the backend)

45
Use Patterns and Motivation Findings
  • Power-of-Now
  • Camera phones are always available
  • Take pictures of ad-hoc subjects
  • Phototaking happens in daily lived context
  • Currently Not Search-Centric
  • For our users, sharing and browsing are more
    important than search and retrieval
  • Role of the Desktop
  • A desktop component adds great value to the
    mobile application by easing browsing
  • Challenges in Offloading Media
  • Non-MMM users find it difficult and frustrating
    to try to get their photos off their cameraphone

46
Use Patterns and Motivation Findings
  • Funnel Effect Selective metadata annotation
  • Two types of annotations
  • Content 1-2 facets describing the photograph
    (e.g., Person, Location)
  • Comment a personal remark about the photograph,
    why they took it, a witty comment, something
    personal to share

47
Use Patterns and Motivation Findings
  • Integrate or motivate annotation
  • Tie annotation to some other activity the user
    already does in a seamless way
  • Sharing
  • Storytelling
  • Motivate users to annotate
  • Demonstrate clear benefit of later reuse
  • Make annotation intrinsically fun (e.g., image
    annotation games)

48
Application Design Findings
  • Input-output application design
  • Combination of simple small applications into
    more powerful applications
  • Network effects
  • Power of many users sharing media and metadata
  • Media and coordination/collaboration
  • Using media to coordinate and collaborate among
    groups
  • Mobile media and games
  • Using media to document games
  • Using games to make media and metadata

49
Nokia Platform Issues
  • Hardware
  • Keypad layout
  • Camera
  • Development environment
  • Access to files
  • Access to camera
  • Access to contextual metadata (time, CellID,
    username)
  • Access to contacts
  • Browser
  • Nokia web browser (caching)
  • Prototyping tools
  • Emulator
  • Wizard of Oz

50
Research Projects
  • Media Streams
  • A framework for creating metadata throughout the
    media production cycle to enable media reuse
  • Active Capture
  • Automates direction and cinematography using
    real-time audio-video analysis in an interactive
    control loop to create reusable media assets
  • Adaptive Media
  • Uses adaptive media templates and automatic
    editing functions to mass customize and
    personalize media
  • Mobile Media Metadata
  • Leverages the spatio-temporal context and social
    community of media capture to automate metadata
    creation for mobile media
  • Social Uses of Personal Media
  • Analysis of social uses of media to predict
    future uses and shape the design of
    next-generation personal media devices and
    applications

51
Social Uses of Personal Photos
  • Looking not just at what people do with digital
    imaging technology, but why they do it
  • Goals
  • Identify social uses of photography to predict
    resistances and affordances of next generation
    mobile media devices and applications
  • Methods
  • Situated video interviews
  • Review of online photo sites
  • Sociotechnological prototyping (magic thing,
    technology probes)

52
Preliminary Findings
  • Social uses of personal photos
  • Memory
  • Creating and maintaining relationships
  • Self-expression
  • Media and resistance
  • Materiality
  • Orality
  • Storytelling

53
From MMM-1 To MMM-2
  • MMM-1 asked
  • What did I just take a picture of?
  • MMM-2 adds
  • Who do I want to share this picture with?

Content
Community
Context
Community
54
MMM-2 Goals
  • Integrated photo sharing and mobile media
    metadata application
  • Metadata as by product of sharing
  • Metadata enables new types of sharing
  • Advanced spatio-temporal-social inferencing
  • Integrated media analysis functions
  • RDF backend for ontology and metadata sharing and
    reuse
  • Connected handset and web applications

55
Sharing ? Metadata
  • From sharing to metadata
  • A birdwatcher takes a photo in a bird sanctuary
    and sends it to her birdwatching group
  • What is the photo of?
  • From metadata to sharing
  • A parent takes a photo of his child on the
    childs birthday
  • Who does he share it with?

56
MMM-2 Use Scenarios
  • Share photo
  • Rank share lists
  • Add/delete members of share list
  • Create new share list
  • Annotate photo
  • Annotate photo on handset
  • Annotate photo on web
  • Annotate photo while sharing on handset
  • Annotate photo while sharing on web

57
Scaling Up Photo Sharing
100K
100M
58
MMM-2 Prototype
  • Phase 1 MMM-2 Prototype Design
  • In Process Spring 2004
  • Phase 2 MMM-2 Prototype Development
  • Summer 2004
  • Phase 3 MMM-2 Prototype Deployment
  • Fall 2004
  • Phase 4 MMM-2 Prototype Evaluation
  • Fall 2004 Spring 2005

59
MMM Publications
  • CHI 2004
  • Anita Wilhelm, Yuri Takhteyev, Risto Sarvas,
    Nancy Van House, Marc Davis. "Photo Annotation on
    a Camera Phone." In Proceedings of the
    Conference on Human Factors in Computing Systems
    (CHI 2004) in Vienna, Austria. ACM Press, 2004.
  • MobiSys 2004
  • Risto Sarvas, Erick Herrarte, Anita Wilhelm, and
    Marc Davis. Metadata Creation System for Mobile
    Images. In Proceedings of the Second
    International Conference on Mobile Systems,
    Applications, and Services (MobiSys2004) in
    Boston, Massachusetts. ACM Press, 2004.
  • Marc Davis, Nathan Good, and Risto Sarvas. "From
    Context to Content Leveraging Context for Mobile
    Media Metadata (Workshop Paper)." Presented At
    MobiSys 2004 Workshop on Context Awareness at the
    Second International Conference on Mobile
    Systems, Applications, and Services in Boston,
    Massachusetts, 2004.
  • Marc Davis. "Mobile Media Metadata (Video)."
    Presented At Second International Conference on
    Mobile Systems, Applications, and Services
    (MobiSys 2004) in Boston, Massachusetts, 2004.

60
MMM Publications
  • ICME 2004
  • Marc Davis and Risto Sarvas. Mobile Media
    Metadata for Mobile Imaging. In Proceedings of
    IEEE International Conference on Multimedia and
    Expo (ICME 2004) Special Session on Mobile
    Imaging in Taipei, Taiwan, IEEE Computer Society
    Press, 2004.
  • ACM Multimedia 2004
  • Marc Davis, Simon King, Nathan Good, and Risto
    Sarvas. "From Context to Content Leveraging
    Context to Infer Media Metadata." In Proceedings
    of 12th Annual ACM International Conference on
    Multimedia (MM 2004) Brave New Topics Session on
    "From Context to Content Leveraging Contextual
    Metadata to Infer Multimedia Content" in New
    York, New York, ACM Press, Forthcoming 2004.
  • Marc Davis. "Mobile Media Metadata Metadata
    Creation System for Mobile Images (Video)." In
    Video Proceedings of 12th Annual ACM
    International Conference on Multimedia in New
    York, New York, ACM Press, Forthcoming 2004.
  • Marc Davis. "Mobile Media Metadata Metadata
    Creation System for Mobile Images (Video
    Description)." In Video Proceedings of 12th
    Annual ACM International Conference on Multimedia
    in New York, New York, ACM Press, Forthcoming
    2004.

61
http//garage.sims.berkeley.edu
About PowerShow.com