Natural Language driven Image Generation - PowerPoint PPT Presentation


PPT – Natural Language driven Image Generation PowerPoint presentation | free to view - id: 6ec873-NzY5N


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation

Natural Language driven Image Generation


Natural Language driven Image Generation Prepared by: Shreya Agarwal Guide: Mrs. Nirali Nanavati Introduction Natural Language driven Image Generation, as the name ... – PowerPoint PPT presentation

Number of Views:81
Avg rating:3.0/5.0
Slides: 47
Provided by: Shr112


Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: Natural Language driven Image Generation

Natural Language driven Image Generation
  • Prepared by Shreya Agarwal
  • Guide Mrs. Nirali Nanavati

  • Natural Language driven Image Generation, as the
    name suggests, refers to the task of mapping a
    natural language text to a scene.
  • The general processes involved in achieving this
    task are
  • Natural Language Understanding
  • Image Retrieval and Positioning

Natural Language Understanding
  • Natural Languages are those used by humans to
    communicate with each other on a daily basis.
    Example English
  • Computers cannot understand Natural Language
    unless it is parsed and represented in a
    predefined template-like form.

Image Retrieval and Positioning
  • This part of the process involves retrieving
    images from the local database or the internet
    relating to the text.
  • The final task is to position the images in a
    manner such that all elements are in their
    correct places in accordance with the natural
    language text.

Systems and Techniques
  • NALIG (NAtural Language driven Image Generation)
  • Text-to-Picture Synthesis Tool 2
  • WordsEye 3
  • Carsim 4
  • Suggested Technique

  • Generated images of static scenes
  • Proposes a theory for equilibrium and stability.
  • Based on description in the form of the following
  • ltsubjectgt ltprepositiongt ltobjectgt
  • ltreferencegt

NALIG Object Taxonomy and Spatial Primitives
  • Defines primitive relationships
  • Example, H_SUPPORT(a,b)
  • Attributes like FLYING, REPOSITORY, etc.
    associated with each object
  • Conditions like CANFLY are used.
  • Example, the airplane on the desert vs. the
    airplane on the runaway

NALIG Object Instantiation
  • All objects mentioned in natural language text
    are initialized.
  • If existence of an object depends on another one,
    it is also instantiated.
  • Such dependence is stored in relation HAS(a,b)
    which defines the strict relationship.
  • Example, branch blocking the window

NALIG Consistency Checking and Qualitative
  • Rules known as naïve statics are defined to
    check for equilibrium and stability. Example,
  • Law of gravity is checked.
  • Space conditions are checked. (Object
    Positioning) Example, The book is on the table.

NALIG Advantages
  • Successful for limited static scene generation
  • Checks equilibrium, space and stability
  • Instantiates implied objects

NALIG Limitations
  • Works for predefined form of phrases. Not
    suitable for full-blown natural language texts
  • Fails to construct dynamic scenes
  • Low success rate for complex scenes

Text-to-Picture Synthesis Tool
  • The technique has the following processes
  • Selecting Keyphrases
  • Selecting Images
  • Picture Layout
  • Example

Selecting Keyphrases
  • Uses keyword-based text summarization
  • Keywords and Phrases extracted based on
    lexicosyntactic rules
  • Unsupervised learning approach based on TextRank
  • Stationary Distribution of Random walk used to
    determine relative importance of words.

Selecting Images
  • Two sources are used in the search for images for
    the selected keyphrases
  • Local database of images
  • Internet based image search engine
  • 15 images retrieved and image processing is done
    to find the correct image.

Picture Layout
  • The technique aims to convey the gist of the
    text. Hence, a good layout is characterized as
  • Minimum Overlap
  • Centrality
  • Closeness
  • A Monte Carlo Randomized algorithm is used to
    solve this highly non-convex optimization problem

  • Successfully conveys the gist of the natural
    language text
  • Searches for images online, thus delivering an
    output for every natural language input
  • Capable of processing complex sentences
  • Fit to represent action sequences

  • Does not render a cohesive image
  • Does not work well for all inputs without a
    healthy internet connection
  • Slower than other methods as it spends time on
    generating a TextRank graph and a co-occurrence

  • This system generates a high quality 3D image
    from a natural language description.
  • It utilizes a large database of 3D models and

WordsEye Linguistic Analysis
  • Utilizes a Part-of-Speech (POS) tagger and a
    statistical parser to generate a Dependency
    Representation of the input text.
  • For Example,

WordsEye Linguistic Analysis
  • This Dependency Representation is then converted
    into a Semantic Representation.
  • It describes the entities in the scene and the
    relations between them.

WordsEye Semantic Representation
  • WordNet is used to find relations between
    different words.
  • Personal names are mapped to male/female humanoid
  • Spatial propositions are handled by semantic
    functions which look at the dependents and
    generate semantic representation accordingly.

WordsEye Depictors
  • Depictors are low-level graphical
    specifications used to specify scenes.
  • They control 3D object visibility, size,
    position, orientation, surface color and
  • They are also used to specify poses, control
    Inverse Kinematics (IK) and modify vertex
    displacements for facial expression.

WordsEye Models
  • Models are stored in the database and have the
    following associated information
  • Skeletons
  • Shape Displacements
  • Parts
  • Color Parts
  • Opacity Parts
  • Default Size
  • Functional Properties
  • Spatial Tags

WordsEye Prepositions denote the layout
  • If we say The daisy is in the test tube, the
    system finds the cup tag for the test tube and
    the stem tag for the daisy. Hence, it puts the
    stem into the cupped opening of the test tube.

WordsEye Poses
  • Poses are used to depict a character in a
    configuration which suggests a particular action
    being performed.
  • They are categorized here as
  • Standalone pose
  • Specialized Usage pose
  • Generic Usage pose
  • Grip pose
  • Bodywear pose

WordsEye Pose examples
  • Specialized Usage pose (Cycling)
  • Grip pose
  • (hold wine bottle)
  • Generic Usage pose
  • (throw small object)

WordsEye Depiction Process
  • Process to convert high level semantic
    representation into low-level depictors.
  • Consists of the following tasks
  • Convert semantic representation from the node
    structure to a list of typed semantic elements
    where all references have been resolved
  • Interpret the semantic representation
  • Assign depictors to each semantic element

WordsEye Depiction Process
  • Resolve implicit and conflicting constraints of
  • Read in the referenced 3D models
  • Apply each assigned depictor to incrementally
    build up the scene while maintaining constraints.
  • Add background environment, ground plane, lights.
  • Adjust the camera (automatically or by hand)
  • Render

WordsEye Depiction Rules
  • Many constraints and conditions are applied so as
    to generate a coherent scene.
  • Constraints are explicit and implicit.
  • Sentences which cannot be depicted are handled by
    using one of Textualization, Emblematization,
    Characterization, Conventional Icons or

WordsEye Advantages
  • Generates high quality 3D models
  • Ability to read poses and grips, constraints and
    use of IK makes the picture coherent.
  • Depiction rules help in mapping linguistically
    analyzed text to exact depictors.
  • Semantic representation lets the depiction
    process truly understand what is being conveyed.

WordsEye Limitations
  • Works on high quality 3D models, hence, required
    a lot of memory and fast searching algorithm.
  • Because of its restriction to its own database,
    the system does not guarantee an output for all
    natural language text inputs.

  • Developed to convert text descriptions of road
    accidents into 3D scenes
  • 2-tier architecture communicating with a formal
    representation of the accident.

Carsim Formalism
  • The tabular structure generated after parsing the
    natural language text has the following
  • Location of accident and configuration of roads
  • List of road objects
  • Event chains for object and movements
  • Collision description

Carsim Information Extraction Module
  • Utilizes tokenizing, part-of-speech tagging,
    splitting into sentences, detecting noun groups,
    named entities, non-recursive clauses and
    domain-specific multiwords for
  • Detecting the participants
  • Marking the events
  • Detecting the roads

Carsim Scene Synthesis and Visualization
  • The previously generated template is taken as
  • Rule-based modules are used to check consistency
    of the scene.
  • A planner is used to generate vehicle
  • A temporal module is used to assign time
    intervals to all segments of these trajectories

Suggested Technique
  • This technique is a hybrid of the techniques we
    have seen so far along with a few additions.
  • It is a theoretical technique and has not been
    implemented yet.

Natural Language Understanding
  • Words of interest will be categorized into the
    following groups using a part-of-speech (POS)
    tagger and a named entity recognizer (NER).
  • SIZE

The template and the co-relation matrix
  • A co-relation matrix specifies position of each
    object in the scene with respect to every other
  • The template for each object in the list of
    objects to be instantiated contains the following
  • Size
  • Co-ordinates
  • Image Location

Image Selection Module
  • This module finds images using two sources
  • Internal database of images
  • Internet based image search engines
  • First 10 images are retrieved
  • Image processing is used to find the correct
  • This image is stored in the database for future

Position Determiner and Synthesis Module
  • The Position Determiner computes the co-ordinates
    of each and every image that is to be placed
    based on the input template (which has the image
    size and location paths).
  • The synthesis module resizes all images and
    places them at the co-ordinates in the template
    (supplied by the position determiner module)

Introducing Machine Learning
  • The aim is to finally make a computer think like
    a human. We can greatly enhance our system by
    using the techniques of Machine Learning.
  • The system can be made to learn the objects
    through unsupervised learning (clustering).
  • The system can be feedback controlled and let
    the user point out meanings of terms (SIZE,
    RELATIVITY, STATE) not previously known.

  • Linguistic analysis is efficient since there is
    no statistical/rule-based parser being used.
  • Searching for images on the internet ascertains
    that an image is generated for every natural
    language input.
  • Introducing machine learning makes the system
    coachable (also, user feedback and instant

  • It might not generate coherent images for complex
    sentences since we do not make use of an advanced
    NLU technique.
  • It depends on internet availability for finding
    images not within its local database.

  • All the methods that have been developed till
    date for tackling the problem have been
  • A technique based on some additions and the
    positives of the existing techniques has been
  • Lot of research is still required to make a
    computer achieve this task as simply as a human
    brain does.

  • 1 ACL 1984 Proceedings of the 10th
    International Conference on Computational
    Linguistics, Natural Language driven Image
    Generation, Giovanni Adorni, Mauro Di Manzo,
    Fausto Giunchiglia, University of Geneo
  • 2 A Text-to-Picture Synthesis System for
    Augmenting Communication, Xiaojin Zhu, Andrew B.
    Goldberg, Mohamed Eldawy, Charles R. Dyer,
    Bradley Strock, University of Wisconsin, Madison
  • 3 Proceedings of the 28th annual conference on
    Computer Graphics and interactive techniques
    2001, WordsEye An Automatic Text-to-Scene
    Conversion System, Bob Coyne, Richard Sproat,
    ATT Labs (Research)
  • 4 Converting Texts of Road Accidents into 3D
    Scenes, Richard Johansson, David Williams, Pierre
    Nugues, 2004 TextMean Proceedings

Thank You! ?