Collaboration Grid work at Anabas and Community Grids Laboratory - PowerPoint PPT Presentation

Loading...

PPT – Collaboration Grid work at Anabas and Community Grids Laboratory PowerPoint presentation | free to download - id: 14ed3b-MDJhM



Loading


The Adobe Flash plugin is needed to view this content

Get the plugin now

View by Category
About This Presentation
Title:

Collaboration Grid work at Anabas and Community Grids Laboratory

Description:

Collaboration Grid work at Anabas and Community Grids Laboratory – PowerPoint PPT presentation

Number of Views:54
Avg rating:3.0/5.0
Slides: 65
Provided by: gridsUcs
Category:

less

Write a Comment
User Comments (0)
Transcript and Presenter's Notes

Title: Collaboration Grid work at Anabas and Community Grids Laboratory


1
Collaboration Grid work at Anabas and Community
Grids Laboratory
  • Indiana University
  • July 30 2007
  • Geoffrey Fox, Marlon Pierce
  • Computer Science, Informatics, Physics
  • Community Grids Laboratories
  • Indiana University Bloomington IN 47404
  • Rui Wang, Alex Ho, Geoffrey Fox
  • Anabas Inc
  • Bloomington, San Francisco
  • gcf_at_indiana.edu
  • http//www.infomall.org

1
2
Community Grids LaboratoryTechnology Expertise
  • Web Service and Web 2.0 technologies for Broad
    Grids
  • Open Grid Forum Web Service architectures
  • Integrate ideas in Flickr Connotea Slideshare
    Youtabe into large scale systems
  • Need to build Broad Grids of Narrow Grids
    (Systems of systems)
  • Geographical Information Systems in Grids
  • Streaming Sensor data (including audio-video
    streams)
  • Portals
  • Multicore parallel computing

3
Community Grids Laboratory
  • Funded by NSF NASA NIH DoE and DoD
  • Cheminformatics High Throughput Screening data
    and filtering PubChem PubMed including document
    analysis
  • Interactive Physics Data Analysis
  • Earthquake Science
  • Sensor Grid GPS global positioning system
  • eSports collaboration for real time trainers and
    sportsman with HPER IU School of Health, Physical
    Education, and Recreation.
  • Ice Sheet Dynamics melting of Glaciers
  • Navajo Nation Grid Education (Science Gateways)
    and Healthcare
  • Web 2.0 tutorial and distance education course
    spring 2007
  • Minority Outreach working with national
    organizations representing 335 Minority Serving
    Universities/Colleges
  • eScience VP for Open Grid Forum

4
Anabas
  • Collaboration Systems (Impromptu)
  • Similar in goal to Webex but with scalable
    event-based architecture using publish-subscribe
    model
  • Works with Community Grids Laboratory on uses of
    Grids in DoD
  • Analysis of Net Centric Operations (NCOW)
  • Analysis of FLTC
  • SBIR shifted from Grid Information systems to
    Sensor Grids based around a network of Grid
    agents
  • Automate construction of Grid from library of
    services and dynamically discovered services
  • Fault Tolerant operation

5
Essential Ideas
  • Distributed software systems are being
    revolutionized by developments from e-commerce,
    e-Science and the consumer Internet. There is
    rapid progress in technology families termed Web
    services, Grids and Web 2.0
  • Many of these developments have important
    implications for collaboration both in terms of
    core technology and capabilities
  • The emerging picture is of distributed services
    with advertised interfaces but opaque
    implementations communicating by streams of
    messages over a variety of protocols
  • Complete systems are built by combining either
    services or Grids (predefined/pre-existing
    collections of services) together to achieve new
    capabilities

6
The Three Technology Families
  • Web Services have clearly defined protocols
    (SOAP) and a well defined mechanism (WSDL) to
    define service interfaces
  • There is good .NET and Java support
  • The so-called WS- specifications provide a rich
    sophisticated standard set of capabilities for
    security, fault tolerance, meta-data, discovery,
    notification etc.
  • Web Service (OGF) Grids build on Web Services and
    provide a robust managed environment with growing
    adoption in Enterprise systems and distributed
    science (so called e-Science)
  • Web 2.0 supports a similar architecture to Web
    services but has developed in a more chaotic but
    remarkably successful fashion with a service
    architecture with a variety of protocols
    including those of Web and Grid services
  • Over 350 Interfaces defined at http//www.programm
    ableweb.com/apis
  • Web 2.0 also has many well known capabilities
    with Google Maps and Amazon Compute/Storage
    services of clear general relevance to DoD
  • There are also Web 2.0 services supporting novel
    collaboration modes as seen in social networking
    sites, portals, MySpace, YouTube,

7
The three service technologies and DoD
  • The Web Service, OGF Grid and Web 2.0 approaches
    differ in important detail but their broad
    architectures are similar and so it is possible
    to use them all in DoD applications
  • We expect growing support with rich functionality
    for all three technology approaches and this plus
    the broad interoperability enabled by a service
    architecture, has important implications for
    capabilities and ease of maintenance and upgrade
    for DoD systems built on these broad-based
    service technologies
  • Anabas analyzed in detail the Net Centric NCOW
    specifications and showed how they could be
    mapped into Web and Grid services
  • This included the NCOW Core Enterprise Services
    and also Sensor Grids and the NCOW Data Model
  • Anabas also addressed how one could achieve
    managed consistent architecture with the
    intrinsically distributed architecture

8
Anabas SBIR Approach
  • This is in collaboration with Community Grids
    Laboratory at Indiana University
  • We follow the OGF Grid architecture and use Web
    services for all capabilities if one needs a
    capability like Google Maps from Web 2.0 it is
    wrapped as a Web Service (and in fact to use an
    Open Geospatial Consortium Interface)
  • We use the powerful open source publish-subscribe
    messaging NaradaBrokering environment to provide
    collaboration (via software overlay networks) and
    fault-tolerance
  • The same software is used to support both Web
    Service messaging (TCP) and audio-video
    conferencing (UDP)
  • We package collections of services as Grids which
    provide particular composite capability such as
    hosting a sensor Grid or supporting one or more
    collaboration functions
  • We are improving core Anabas collaboration
    technology to support shared video not supported
    well by Webex
  • We provide Grid Builder tool to build Grids by
    composing other Grids together and to dynamically
    manage them
  • We provide sensor Grid architecture and will
    demonstrate with many types of sensors

9
Comparison of Web 2.0 and Grids
  • See http//grids.ucs.indiana.edu/ptliupages/presen
    tations/CTSpartIMay21-07.ppt

10
Architecture of Streaming Grids of Grids
  • And describing the underlying messaging system
    NaradaBrokering and how message multicast enables
    collaboration

11
Raw Data ? Data ? Information ?
Knowledge ? Wisdom
AnotherGrid
Decisions
AnotherGrid
SS
SS
SS
SS
FS
FS
OS
MD
MD
FS
Portal
Portal
FS
OS
OS
OS
SOAP Messages
OS
FS
FS
FS
FS
AnotherService
FS
MD
MD
OS
MD
OS
OS
FS
Other Service
FS
FS
FS
FS
OS
MD
OS
OS
FS
FS
FS
MD
MD
FS
Filter Service
OS
FS
MetaData
AnotherGrid
FS
FS
FS
MD
Sensor Service
SS
SS
SS
SS
SS
SS
SS
SS
SS
SS
AnotherService
12
Grid Service Philosophy I
  • Services receive data in SOAP messages,
    manipulate it and produce transformed data as
    further messages
  • Knowledge is created from information by services
  • Information is created from data by services
  • Semantic Grid comes from building metadata rich
    systems of services
  • Meta-data is carried in SOAP messages
  • The Grid enhances Web services with semantically
    rich system and application specific management
  • One must exploit and work around the different
    approaches to meta-data (state) and their
    manipulation in Web Services

13
Grid Service Philosophy II
  • There are a horde of support services supplying
    security, collaboration, database access, user
    interfaces
  • The support services are either associated with
    system or application where the former are WS-
    and GS- which implicitly or explicitly define
    many support services
  • There are generalized filter services which are
    applications that accept messages and produce new
    messages with some data derived from that in
    input
  • Simulations (including PDEs and reactive
    systems)
  • Data-mining
  • Transformations
  • Agents
  • Reasoning are all termed filters
    here
  • Agent Systems are a special case of Grids
  • Peer-to-peer systems can be built as a Grid with
    particular discovery and messaging strategies

14
Grid Service Philosophy III
  • Filters can be a workflow which means they are
    just collections of other simpler services
  • Grids are distributed systems that accept
    distributed messages and produce distributed
    result messages
  • A service or a workflow is a special case of a
    Grid
  • A collection of services on a multi-core chip is
    a Grid
  • Sensors or Instruments are managed by services
    they may accept non SOAP control messages and
    produce data as messages (that are not usually
    SOAP)
  • Collaborative services share either input
    (replicated model) or output ports
  • Collaboration involves a sharing messaging system
    (naturally publish-subscribe) and a control
    formalism (XGSP is SOAP compatible H323/SIP)

15
Raw Data ? Data ? Information ?
Knowledge ? Wisdom
AnotherGrid
Decisions
AnotherGrid
SS
SS
SS
SS
FS
FS
OS
MD
MD
FS
Portal
Portal
OS
OS
OS
FS
SOAP Messages
OS
FS
FS
FS
FS
AnotherService
FS
MD
MD
OS
OS
OS
Other Service
FS
MD
FS
FS
FS
FS
FS
OS
MD
OS
OS
FS
FS
FS
FS
MD
FS
Filter Service
OS
FS
MetaData
AnotherGrid
FS
FS
FS
MD
Sensor Service
SS
SS
SS
SS
SS
SS
SS
SS
SS
Collaboration by Message Replication
AnotherService
16
Master
Event(Message)Service
OtherParticipants
Shared Input Port (Replicated WS) Collaboration
with UFIOas User Facing and SFIO as Service
Facing Ports
17
Master
WS Display
WS Viewer
Event(Message)Service
OtherParticipants
WSDisplay
WS Viewer
Shared Output Port (Single WS) Collaboration
that Can be shared at any point on visualization
pipeline
18
Raw Data ? Data ? Information ?
Knowledge ? Wisdom
AnotherGrid
Decisions
AnotherGrid
SS
SS
SS
SS
FS
FS
OS
MD
MD
FS
Portal
Portal
FS
OS
OS
OS
SOAP Messages
OS
FS
FS
FS
FS
AnotherService
FS
MD
MD
OS
OS
FS
Other Service
FS
FS
FS
FS
MD
OS
MD
OS
Portal
OS
FS
FS
MD
MD
FS
Filter Service
OS
FS
MetaData
AnotherGrid
FS
FS
FS
MD
Sensor Service
SS
SS
SS
SS
SS
SS
SS
SS
SS
Collaboration by Message Replication at any point
infilter chain
SS
Shared Display is the last filter
AnotherService
19
(No Transcript)
20
NaradaBrokering 2003-2006
  • Messaging infrastructure for collaboration,
    peer-to-peer and Grids Implements JMS and native
    high-performance protocols (message transit time
    of 1 to 2 ms per hop)
  • Order-preserving message transport with QoS and
    security profiles
  • Support for different underlying transport such
    as TCP, UDP, Multicast, RTP
  • SOAP message support and WS-Eventing, WS-RM and
    WS-Reliability.
  • Active replay support Pause and Replay live
    streams.
  • Stream Linkage can link permanently multiple
    streams using in annotation of real-time video
    streams
  • Replicated storage support for fault tolerance
    and resiliency to storage failures.
  • Management Scripting Interface to streams and
    brokers (uses WS-Management) for initialization,
    firewall issues and fault tolerance
  • Broker Topics and Message Discovery Locate
    appropriate
  • High Performance Transport supporting SOAP
    Infoset for GIS applications

21
These measurements are messages from client to
broker and back using latest Java 1.6 release
that is about twice performance of earlier
releases
22
These measurements are messages from client to
broker and back using latest Java 1.6 release
that is about twice performance of earlier
releases. This graph is identical to previous one
for small messages
23
Average Video Delays UDP Performance when
NaradaBrokering used for audio-video conferencing
24
Broad Grid of Narrow Grids
25
Grid Builder and Dynamic Sensor Grids
  • Separate Talk by Rui Wang

26
What is a Simple Service?
  • Take any system it has multiple functionalities
  • We can implement each functionality as an
    independent distributed service
  • Or we can bundle multiple functionalities in a
    single service
  • Whether functionality is an independent service
    or one of many method calls into a glob of
    software, we can always make them as Web
    services by converting interface to WSDL
  • Simple services are gotten by taking
    functionalities and making as small as possible
    subject to rule of millisecond
  • Distributed services incur messaging overhead of
    one (local) to 100s (far apart) of milliseconds
    to use message rather than method call
  • Use scripting or compiled integration of
    functionalities ONLY when require lt1 millisecond
    interaction latency
  • Apache web site has many (pre Web Service)
    projects that are multiple functionalities
    presented as (Java) globs and NOT (Java) Simple
    Services
  • Makes it hard to integrate sharing common
    security, user profile, file access .. services

27
Grids of Grids of Simple Services
  • Link via methods ? messages ? streams
  • Services and Grids are linked by messages
  • Internally to service, functionalities are linked
    by methods
  • A simple service is the smallest Grid
  • We are familiar with method-linked
    hierarchyLines of Code ? Methods ? Objects ?
    Programs ? Packages

28
Component Grids?
  • So we build collections of Web Services which we
    package as component Grids
  • Visualization Grid
  • Sensor Grid
  • Utility Computing Grid
  • Collaboration Grid
  • Earthquake Simulation Grid
  • Control Room Grid
  • Crisis Management Grid
  • Drug Discovery Grid
  • Bioinformatics Sequence Analysis Grid
  • Intelligence Data-mining Grid
  • We build bigger Grids by composing component
    Grids using the Service Internet

29
Using the Grid of Grids and Core Services to
build multiple application grids re-using common
components.
BioInformatics Grid
Chemical Informatics Grid


15 Application Services Sequencing
Tools Biocomplexity Simulations
Domain Specific Grids/Services
15 Application Services Screening Tools Quantum
Calculations
14 Information
Instrument/Sensor
11 Portals
Services
13 Data Access/Storage
12 Computing
17 Collaboration
9 Management 18 Scheduling
10 Policy
4 Notification
8Metadata
7 Discovery
Core Low Level Grid Services
5 Workflow
6 Security
3 Messaging
9 Management
Physical Network (monitored by FS16)

30
Net Centric and Critical Infrastructure (CI)
Grids built as Grids of Grids and re-using
subGrids
31
Mediation and Transformation in a Grid of Grids
and Simple Services
32
Technology Nuggets produced for Collaboration
Grids
  • Group Support in Anabas Collaboration Framework
  • Hybrid Shared Display
  • GlobalMMCS is a collaboration system built using
    services and publish-subscribe messaging
  • Improved Java Media Framework

33
Collaborative Groups Illustrated In Anabas
Impromptu
  • Examples of applications
  • private discussions in conference/lecture
  • simultaneous breakout groups
  • Multiple broadcasting in the same session (e.g.
    audio/voice or video/TV channels for
    user-defined, such as particular need-to-know,
    groups)

34
Group Sharedlets
  • An Anabas Sharedlet is a shared application,
    e.g. TextChat, VoIP, Video Conferencing, Shared
    Applications, Whiteboard
  • GroupManager provides preliminary Group
    information to each sharedlet, include joined
    sessions, active session, session participants,
    participant privileges (e.g. host, presenter) in
    each session
  • Each Sharedlet has its own specific method to
    handle Group. E.g.
  • Text Sharedlet stores all conversations in every
    sessions
  • Video Sharedlet displays the videos in the active
    session only
  • Audio Sharedlet plays the audio in the active
    session only
  • Shared Display Sharedlet may store data in every
    sessions or in the active session only
  • The Sharedlet specific method depends on network
    bandwidth requirement (e.g. Is the network
    bandwidth sufficient?) and usage difference (e.g.
    Can past data be disposed? Who can share
    information?)

35
Video
Group
Text
Audio
36
HSD Hybrid Shared Display
  • HSD builds on a combination of Classic Shared
    Display (CSD) and Video Shared Display (VSD)
  • Problem Video sharing using lossless encoding
    scheme consumes very high network bandwidth
  • Motivation of HSD Find the video or fast
    changing regions in the shared application, and
    encode them using video codec e.g. H.261 and
    MPEG4 to save network bandwidth while retaining
    good visual quality

37
(No Transcript)
38
Illustration of Hybrid Shared Display on the
sharing of a browser window with a fast changing
region.

39
HSD Flow
Screen capturing
Region finding
VSD
CSD
Video encoding
SD screen data encoding
Presenter
Through NaradaBrokering
Network transmission (RTP)
Network transmission (TCP)
Participants
Video Decoding (H.261)
SD screen data decoding
Rendering
Rendering
Screen display
40
GlobalMMCS Web Service Architecture
Use Multiple Media servers to scale to many
codecs and many versions of audio/video mixing
WebServices
High Performance (RTP)and XML/SOAP and ..
NB Scales asdistributed
Gateways convert to uniform XGSP Messaging
NaradaBrokering
41
Global-MMCS Community Grid
  • This includes an open source protocol independent
    Web Service MCU which will scale to an
    arbitrary number of users and provides support
    for thousands of simultaneous users of
    collaboration services.
  • The function of A/V media server is distributed
    using NaradaBrokering architecture.
  • Media Servers mix and convert A/V streams
  • Open XGSP MCU based on the following open source
    projects
  • openh323 is basis of H323 Gateway
  • NIST SIP stack is basis of SIP Gateway
  • NaradaBrokering is open source messaging
  • Java Media Framework basis of Media Servers
  • Helix Community http//www.helixcommunity.org for
    Real Media
  • http//www.globalmmcs.org open source release

42
Break up into Services
  • Monolithic MCU becomes many different Simple
    Services
  • Session Control
  • Thumbnail image grabber
  • Audio Mixer
  • Video Mixer
  • Codec Conversion
  • Helix Real Streaming
  • PDA Conversion
  • H323/SIP Session/Signaling Gateways
  • As independent can replicate particular services
    as needed
  • Codec conversion might require 20 services for 20
    streams spread over 5 machines
  • 1000 simultaneous users could require
  • 1 session controller, 1 audio mixer, 10 video
    mixers, 20 codec converters, 2 PDA converters and
    20 NaradaBrokers
  • Support with a stream optimized Grid Farm in the
    sky
  • Future billion way Video over IP serving 3G
    Phones and home media centers/TVs could require
    a lot of computing

43
Collaboration Grid
XGSP Media Service
WS-Context
NaradaBroker
Audio Mixer
HPSearch
Video Mixer
UDDI
NaradaBroker
Transcoder
Thumbnail
WS-Security
Replay
NaradaBroker
Record
Annotate
SharedWS
SharedDisplay
WhiteBoard
44
GlobalMMCS and NaradaBrokering
  • All communication both control and binary
    codecs are handled by NaradaBrokering
  • Control uses SOAP and codecs use RTP transport
  • Each stream is regarded as a topic for NB
  • Each RTP packet from this stream is regarded as
    an event for this topic
  • Can use replay and persistency support in NB to
    support archiving and late clients
  • Can build customized stream management to
    administer replay, and who gets what stream in
    what codec
  • NaradaBrokering supports unicast and multicast
  • Use firewall penetration and network monitoring
    services in NB to improve Q0S

45
XML based General Session Protocol XGSP
  • The XGSP conference control includes three
    services
  • Conference management
  • supports user sign-in, user
    create/terminate/join/leave/invite-into XGSP
    conferences
  • conference calendar service
  • Application session management
  • provides users with the service for
    creating/terminating application sessions,
    managing session related services such as
    audio/video mixing
  • Floor control
  • manages the access to shared collaboration
    resources in different application sessions
  • for example, in a large scale of meetings
    having thousands of people, only limited people
    are allowed to become presenters so that they can
    send audio/video

46
Improved Java Media Framework Performance
Video Rendering performance (left still desktop,
right movie sequence) We plot CPU percentage use
versus number of streams rendered
47
GlobalMMCS SWT Client
TV
GIS
Chat
Webcam
Video Mixer
48
Integration of PDA, Cell phone and Desktop Grid
Access
NB Support for optimizedPDA Communication
49
Real time annotation and replay I
50
Snapshot during a recording/annotation session
51
Replay of an annotation session
52
Sensor and GIS Grids
  • See also PhD Thesis http//grids.ucs.indiana.edu/p
    tliupages/publications/GalipAydin-Thesis.pdf
    http//grids.ucs.indiana.edu/ptliupages/presentati
    ons/galip-aydin-defense.ppt
  • Paper http//grids.ucs.indiana.edu/ptliupages/publ
    ications/PEPIRealTimeGISAydin_YB.pdf
  • Separate talk by Marlon Pierce

53
Analysis of DoD Net Centric Services in terms of
Web and Grid services
54
The Grid and Web Service Institutional Hierarchy
4 Application or Community of Interest
(CoI)Specific Services such as Map Services,
Run BLAST or Simulate a Missile
XBMLXTCE VOTABLE CML CellML
3 Generally Useful Services and Features (OGSA
and other GGF, W3C) Such as Collaborate,
Access a Database or Submit a Job
OGSA GS-and some WS- GGF/W3C/.XGSP (Collab)
2 System Services and Features (WS- from
OASIS/W3C/Industry) Handlers like WS-RM,
Security, UDDI Registry
WS- fromOASIS/W3C/Industry
1 Container and Run Time (Hosting) Environment
(Apache Axis, .NET etc.)
Apache Axis.NET etc.
Must set standards to get interoperability
55
The Ten areas covered by the 60 core WS-
Specifications
56
Activities in Global Grid Forum Working Groups
57
Net-Centric Core Enterprise Services
58
Produce the Needed Core Services
  • We can classify services in many ways and
    following 2 charts are one way slightly changed
    from proposal as NCOW and our work changed a
    little.
  • Green is in hand we know a lot
  • Orange is in hand with outside but available
    solutions
  • Red has problems Security does not have
    industry consensus while current Scheduling work
    does not address DoD real-time service and
    network requirements

59
The Core Features/Service Areas I
60
The Core Feature/Service Areas II
61
Some Conclusions I
  • One can map nearly all NCOW/NCES and GiG core
    capabilities into Web Service (WS-) and Grid
    (GS-) architecture and core services
  • Analysis of Grids in NCOW/NCES document
    inaccurate (confuse Grids and Globus and only
    consider early activities)
  • Some mismatches on both NCOW and Grid sides
  • GS-/WS- do not have collaboration and miss some
    messaging
  • NCOW does not have at core level system metadata
    and resource/service scheduling and matching
  • Higher level services of importance include GIS
    (Geographical Information Systems), Sensors and
    data-mining

62
Some Conclusions II
  • Criticisms of Web services in a paper by Birman
    seem to be addressed by Grids or reflect
    immaturity of initial technology implementations
  • NCOW/NCES does not seem to have any analysis of
    how to build their systems on WS-/GS-
    technologies in a layered fashion they do have a
    layered service architecture so this can be done
  • They agree with service oriented architecture
  • They seem to have no process for agreeing to WS-
    GS- or setting other standards for CES
  • Grid of Grids allows modular architectures and
    natural treatment of legacy systems
  • Note Grids, Services and Handlers are all just
    entities with distributed message-based input and
    output interfaces

63
Additional Services
  • Sensors have low level support listed as FS3
    higher level integration using SensorML and
    Filters well understood. Some work in phase I
  • GIS Grid services pioneered by team and already
    shown in phase I
  • Mediation (Interoperability) Services needed to
    link Grids (defined as a collection of 1
    Services)
  • Need to generalize existing solutions for Sensor
    Grids and for MQSeries-SOAP Mediation
  • View NaradaBrokering as a SOAP Intermediary

64
Out of Scope for Phase II
  • Many areas are still evolving significantly
  • Mediation/Interoperation
  • Security
  • Scheduling of non-compute Resources
  • Data/Information Federation
  • Semantic Grid and management
  • We will not test scalability on large number of
    services, sensors and component Grids
  • Integrating legacy systems not addressed
  • Grid of Grids building tool is new idea can
    expect will benefit from further work
About PowerShow.com