Using Automated Task Solution Synthesis to Generate Critical Junctures for Management of Planned and

1
Using Automated Task Solution Synthesis to
Generate Critical Junctures for Management of
Planned and Reactive Cooperation between a
Human-Controlled Blimp and an Autonomous Ground
Robot

• A Thesis Presented for the Master of Science
  Degree
• The University of Tennessee, Knoxville
• Chris Reardon
• July 2, 2008

2
Outline

• Introduction
• Related Work
• Approach
• Experiments
• Evaluation
• Conclusion

3
Introduction
4
Introduction Heterogeneous Cooperative
Multi-robot Systems

• Multi-robot systems have advantages over
  single-robot systems
• Cooperation, distributed tasks, distributed
  resources, parallelism, redundancy
• Heterogeneity advantages over homogeneity
• Cost, specialization, difficulty of true
  homogeneity

5
IntroductionPurpose and Contribution

• Previous work has identified (offline and
  manually) periods of beneficial cooperation in
  multi-robot teams
• Hypotheses
• A human-robot teams performance can be increased
  by cooperating during certain periods of a
  mission
• These periods can be identified algorithmically
• Identification can be performed offline and
  reactively online
• Validate through real-world experimentation

6
Related Work
7
Two Directly Related Works

• H. Choxi, C. Bolden Distributed Control for
  Unmanned Vehicles. Lockheed Martin Advanced
  Technology Laboratories unpublished internal
  paper, 2005.
• F. Tang, L. E. Parker. ASyMTRe Automated
  Synthesis of Multi-Robot Task Solutions through
  Software Reconfiguration. Proceedings of IEEE
  International Conference on Robotics and
  Automation, 2005.

8
Directly Related WorksLockheed Martin ATL

• Choxi and Bolden, 2005
• Use tightly-coupled coordination at critical
  points to increase team effectiveness
• Key concepts
• A team of less-than-fully-capable robots can
  increase effectiveness by sometimes cooperating
• Because the team only cooperates sometimes,
  overall efficiency isnt severely impacted

9
Directly Related WorksLockheed Martin ATL

• Critical Junctures the points in a mission
  where Independent behaviors will fail and
  coordination is required
• Room for improvement
• Experiments conducted only in simulation
• A human identified all of the critical junctures
  and behaviors required manually

10
Directly Related WorksLockheed Martin ATL -
Simulation

• UGV (robot) and UAV (blimp) searching office
  environment for targets, some high, some low
• UGV
• cant see high
• can localize
• UAV
• can only wall-follow, blob-follow, wander
• Noise applied to UAV position to simulate air
  currents

11
Directly Related WorksLockheed Martin ATL
Simulation

• Three strategies
• Independent
• UAV and UGV search separately UAV wall-follows
  and wanders
• Dependent
• UAV uses camera to follow UGV
• Synergistic
• UAV follows UGV at key periods where
  wall-following not available, instead of
  wandering

12
Directly Related WorksLockheed Martin ATL
Simulation

• Synergistic approach
• UAV has difficulty searching cubes on left due to
  noise, lack of walls
• C is the initial critical juncture where
  coordination begins (UAV follows UGV using
  blob-following behavior)
• Left cubicles are searched
• D is the critical juncture where coordination ends

13
Directly Related WorksLockheed Martin ATL
Simulation

• Findings
• When no noise is present, Independent works
• When moderate to high amounts of noise are
  present, synergistic approach balances
  effectiveness with efficiency

14
Directly Related WorksASyMTRe

• Parker and Tang, 2005
• Abstract robot abilities into environmental
  sensors and perceptual, motor, and communication
  schemas
• Autonomously connect schemas at run-time
• Dynamic task solution reconfiguration
• Sharing of information and assistance between
  team members

15
Approach
16
Approach Use ASyMTRe

• Critical junctures
• Points where cooperation begins and ends
• Bound cooperative regions
• Use ASyMTRe to identify critical junctures
• Define task and robot abilities so that there can
  be regions of beneficial cooperation, but not all
  aspects of task require cooperation
• Do this in such a way as to be
• Realistic
• Generalizable

17
Approach Benefits of ASyMTRe

• Benefits of using ASyMTRe
• By identifying CJs algorithmically, the human is
  removed from the process automation
• Because ASyMTRe is capable of online task
  reconfiguration, CJs can be identified reactively
  online

18
Approach Task Selection

• Use an autonomous mobile ground robot (UGV) and a
  human-controlled blimp (UAV)
• Coverage problem Visit all of a space looking
  for targets
• Metrics defined Target Localization Accuracy,
  Task Completion Time, Aggregate Run Time, False
  Positive Rate

19
Approach Defining Information

• Environmentally Dependent Information (EDIs)
• Key Concept In general, aspects of the
  environment supply different information, which
  makes different schemas available at different
  points in the environment

20
Approach EDIs Defined

• Landmark concept
• When the UAV is within a specified range of an
  EDI, localization schemas are available
• Define this through ASyMTRe configuration

21
Approach EDI selection

• Corners as landmark EDIs (blue dots)
• Within 3m (blue shaded circle) UAV can localize

22
Approach Behavior Concepts

• Independent
• Team members perform task without cooperation
• No CJs are identified
• Cooperative Planned
• Use ASyMTRe to identify CJs in advance
• Cooperative Reactive
• Identify CJs during the scenario as needed
• Note Cooperatives equate to Synergistic from
  Choxi and Bolden

23
Approach Some Experiment Details

• Coverage Robot and UAV search for and localize
  targets
• Map of environment provided a priori
• UAV and robot have predefined paths, waypoints
• At waypoints, scan for targets

24
Approach Target Selection

• Color solid, unique, detectable
• Roughly symmetrical
• Size
• larger than small abnormalities
• Small enough to mount up high
• 12 green balloons fit all criteria and were
  inexpensive

25
Approach Environment

• Claxton 2nd floor
• High ceilings
• Wide hallways
• Tables and benches arranged for diverse
  environment
• Test area limited by camera, remote range to
  400-500 sq ft

26
Approach Target Placement

• 12 targets
• Randomly generated positions
• Minimum spacing 4m
• High (over 1m up) or low (on ground)
• Deployment hidden from UAV team

27
Approach Construct a UAV

• Custom construction
• Two propulsion motors (reversible, joined) on
  rotatable axle
• One tail motor
• 2.4 GHz Micro camera
• 6 poly envelope
• 18 cu ft helium 18 oz lift, 17.1 oz weight

28
Approach UAV Abilities

• UAV
• Human-operated using remote, on-board camera
• Navigation
• Target identification
• Localization in range of EDIs
• Relative localization with robot assistance
• Vertical movement

29
Approach UAV camera view

• Eyecam 2.4GHz color micro wireless camera
• 92 fov, f3.6mm
• 320x160 resolution
• 20g weight

30
Approach UAV Piloting Challenges

• UAV Piloting Challenge Movie

31
Approach Robot Abilities

• Robot
• Autonomous, using Player robot control server
• Localization
• Navigation
• Communication
• Target identification and relative localization

32
Approach Robot Camera View

• A target viewed from the robots Canon VC-C4
  camera taken via the Player playercam
  interface.
• Green rectangular overlay over the balloon
  highlights the blob that the blobfinder reports.

33
Approach Robot behavior

• Follow path
• Stop at waypoints (3-4m apart)
• Rotate 360, stopping every 36 to scan for
  targets
• If a possible target is found
• Use target size, shape to confirm it as a target
• Use target size, orientation to localize target
• Target localization range 0.5 3.5m
• Target localization accuracy within 2m

34
Approach Correspondence Problem

• Correspondence Problem
• Which localization corresponds to which actual
  target?
• Causes
• Robot stops and scans at close intervals
• Targets are identical in appearance
• One target can be detected more than once (at
  different waypoints)
• Localization accuracy is not precise (within 2m)
• Decision
• Allow multiple localizations of one target
• Count any localization as correct if within 2m

35
Experiment
36
Experiment EDI selection

• EDIs
• corners of main walls (gt 2m length)
• Easily identifiable
• Common, but not overabundant
• Within 3m (blue shaded circle) UAV can localize

37
Experiment Paths and Cooperative Regions

• Cooperative Regions (boxed areas)
• Approximate area inside of which cooperation is
  necessary for UAV to localize

38
Experiment Target Setup

• 5 maps
• Randomly generated
• 12 targets
• 4m minimum spacing
• Eligible positions 0.5m apart

39
Experiment Target Setup
40
Experiment Setup

• The UAV is flown by a pilot via remote using
  video from the micro cam to navigate
• Co-pilot identifies targets on video, uses
  computer to communicate with robot on behalf of
  UAV
• Stop
• Move to position x,y
• Go

41
Experiment Independent Scenario

• UAV and robot
• Search the same target configuration map for
  targets
• Attempt to localize targets detected
• Operate independently
• Notes
• Robot cannot detect high targets due to fixed
  camera angle (15 down)
• UAV cannot localize targets outside of EDI range

42
Experiment Cooperative Planned Scenario

• Paths of UAV and robot planned to coincide at
  segments where cooperation is required
• CJs identified before mission begins
• Robot and UAV proceed through pre-planned paths
• At each CJ, robot waits for UAV to localize any
  nearby targets and send go message
• Robot and UAV move to next goal position
• Time intensive for UAV

43
Experiment Cooperative Reactive Scenario

• Proceeds like Independent, except
• When UAV finds critical juncture (in real-time)
• UAV operator calls robot to assist in
  localization
• Robot comes to UAV, communicates position
• UAV operator uses robot for relative localization
• UAV operator sends go command to robot
• Robot goes to last position, resumes
• No preplanning required
• Robot can move a great deal

44
Experiment Example

• Robot and UAV begin a Cooperative Planned run.

The UAV and robot navigate the environment.
45
Experiment - Example

• The robot assists the UAV in localizing a target.

Localization is complete the UAV and robot
continue.
46
Experiment - Movies

• Cooperative Planned 1
• Cooperative Planned 2
• Cooperative Reactive 1
• Cooperative Reactive 2

47
Evaluation
48
Evaluation Metrics Examined

• Target Localization Accuracy
• Task Completion Time
• Aggregate Run Time
• False Positive Rate

49
Evaluation Target Localization Accuracy

• Ratio of targets localized to actual targets
• Cooperative behaviors performed identically
• Cooperative behaviors performed better than
  Independent
• Some tasks can only be accomplished through
  cooperation

50
Evaluation Task Completion Time

• Maximum of robot and UAV time per run
• Independent is fastest, followed by Cooperative
  Planned, Cooperative Reactive
• Cooperative Reactives longer run time due to
  unplanned CJs often called to localize by UAV
  when far away
• Benefit of no planning set against longer TCT

51
Evaluation Aggregate Run Time

• Sum of robot and UAV time per run
• Represents approximate energy cost, wear and tear
• Cooperative Planned is so long because even
  though UAV is faster, robot UAV paths must
  coincide
• Cooperative Reactive far superior to Planned

52
Evaluation False Positive Rate

• Ratio of incorrect localizations to reported
  localizations
• Cooperative Reactive approach appears to yield
  more false positives than Planned

53
Evaluation False Positive Rate

• Most false positives due to orientation error
• Reactives poor performance possibly due to
  unsettled state after moving back and forth to
  assist UAV
• Possible solution add a settling behavior
  after each assist

Independent
54
Evaluation False Positive RatePlanned vs.
Reactive

• Planned

Reactive
55
Conclusion
56
Conclusion Discoveries

• Target Localization Accuracy
• Cooperative behaviors superior to Independent
• Task Completion Time
• Planned Cooperative better than Reactive
• Aggregate Run Time
• Reactive Cooperative much better than Planned
• False Positive Rate
• Planned Cooperative appears better than Reactive
• Cooperative behaviors best balance efficiency and
  effectiveness, confirming previous work

57
Conclusion Discoveries
58
Conclusion Recommendations

• Cooperative Reactive
• Best where
• Planned approach not possible
• Energy or wear a consideration
• Significant number of CJs are not expected
• Cooperative Planned
• Best where
• Large number of CJs expected
• False Positive Rate must be minimized
• Task Completion Time must be minimized

59
Conclusion Contributions

• Leveraged the power of ASyMTRe to identify
  critical points at which tightly-coupled
  cooperation/coordination benefits a heterogeneous
  team of less-than-fully-capable robots
• Cooperative regions/Critical Junctures
• Previous related work Choxi Bolden, 2005
  identified these critical points manually
• I have identified them algorithmically
• a priori
• reactively during task execution
• Validation
• Previous related works approach validated in
  simulation
• I have validated the benefit of cooperation at
  these critical junctures in real-world
  experimentation

60
Conclusion Future Work

• Expand EDI concept
• Self-identify EDIs
• Preserve human-interaction while abstracting and
  automating some UAV capabilities
• Scale team size
• Engineering improvements

61
False Positive DiscussionMap 2
Independent
Cooperative Planned
Cooperative Reactive