Developing%20applications%20that%20harness%20the%20Agricultural%20and%20Natural%20Resource%20Grid%20(and%20linking%20your%20own%20data)

1
Developing applications that harness the
Agricultural and Natural Resource Grid (and
linking your own data)

• Matthew Laurenson and Seishi Ninomiya
• National Agricultural Research Center
• Tsukuba
• Japan

2
Contents

• Concepts
• Remote Procedure Calls (RPC)
• Brokers
• Using a broker
• Metadata requests
• Data requests
• Issues
• Linking your own data
• SOAP
• New brokers and future enhancements

3
Overall Goal To help you make portable
software applications (DSS).
4
Current Situation Web-based but

• Applications closely coupled to databases
• Limited range of applications at each site
• Duplicated development at each site

5
Ag and NR Grid a Hotel Room for Agricultural
and Natural Resource DSS
User interface in national language
Room
DSS
DSS
DSS
Soils
Maps
Weather
Elevation
6
New Zealand
Maps
Soils
Elevation
Weather
7
??
??
?
?????
??
8
??
??
???
??
??
9
Goal of Ag and NR GridTo provide this hotel
room in your country, so you can use models
developed by others (and they can use your
models).
10
Distributed Computing Concepts

• Remote Procedure Calls
• Brokers to handle heterogeneous databases

11
Remote Procedure Calls (RPC)

• RPCs let a program execute a procedure on a
  remote computer as if it were local.

12
Remote Procedure Call

• My Computer
• remotefindRemote()
• A5
• B6
• Cremote.add(A,B)
• Print C
• ...

• Remote Computer
• procedure add(x,y)
• return xy
• ...

13
Remote Procedure Calls (RPC)

• RPCs let a program execute a procedure on a
  remote computer as if it were local.
• Newer RPC mechanisms let programs send and
  receive data objects.
• RPC mechanisms CORBA, .NET, Java RMI, SOAP...
• RMI only Java programs can use
• SOAP used by other languages as well

14
RPC vs Local Calls

• Latency (delay)
• RPCs much slower than local calls
• difficult to predict how long call will take
• May fail
• network problems
• problems with other computer (server)
• Security issues
• firewalls...

15
But...

• Can distribute processing and data over multiple
  machines
• Great when multiple users must share the same
  dynamic data item or communicate with each other.

16
Mediated Architecture Brokers
Domain Databases (Heterogeneous)
Consistent data access (using RPC mechanism)
Applications
Broker
17
MetBroker
Weather Databases
Applications
Data Retrieval Servlet
South Africa
Web Browsers
Data Retrieval
Korea
Crop Growth
MetBroker
Japan
Disease Risk
New Zealand
Weather Risk
USA
18
Contents

• Concepts
• Remote Procedure Calls (RPC)
• Brokers
• Using a broker
• Metadata requests
• Data requests
• Issues
• Linking your own data
• SOAP
• New brokers and future enhancements

19
Data and Metadata Requests
Interaction between applications and brokers uses
objects data packages
Metadata request
Standard request
DB-specific query
Broker
Application
Database (various)
Metadata result
DB-specific result
Standard result
Metadata
Data
20
Example - MetBroker Services

• Weather Data
• Single station
• All stations in a region
• Metadata (data about data)
• Database details (coverage of globe, login req.)
• Regions within databases
• Station details (latitude, longitude, start date,
  end date, elements recorded, resolution...)

21
Elements of a Station Data Request
Weather station
Username/password
Time interval
Resolution
Required weather elements
Allow summarizing
22
Send Request to MetBroker and Receive Result
?
Data Result Tsuchiura Operated 1976-present 1999/1
/3 0600-1999/2/1 0000 10.1, -0.2, 0 12.6, 3.8,
0 11.3, -1.5, 0.3 ...
To MetBroker
Data Request amedas 40341 rain, air
temperature 1999/1/1-1999/2/1 daily OK to
summarize
23
Query Results - Single Station
Station query details
Station details and data sequences are
encapsulated in a single result object
rain
air temp.
solar rad.
24
Spatial Request
Client
Broker
Temperature
Rainfall
Radiation
25
Spatial Request
Background map dynamically retrieved from
http//www.mapblast.com
Daily max/min temperatures rainfall for all
stations in region
26
Query results - multiple stations
27
Using Brokers Implementation Issues
28
RMI Blocked by Firewalls
Wrapper Servlet
ClientApplication Java
RMI
HTTP
Broker
Client-side firewalls
RMI
ClientApplication Java
Server Machine
29
Applet Using Multiple Brokers
DEMBroker
Elevation data
MetBroker
ChizuBroker
Background maps
Meteorological data
Spatial Risk Applet
Application text
Resource Server
30
Without Multiple Threads - Sequential
DEMBroker
MetBroker
ChizuBroker
Spatial Risk Applet
Resource Server
12 seconds
31
With Multiple Threads Parallel
DEMBroker
MetBroker
ChizuBroker
Spatial Risk Applet
Resource Server
5 seconds
32
JavaBeans for Java GUI programs

• For rapid GUI development in Java can use
  existing MetBeans
• Transparently handle firewalls, latency...
• Making a simple data retrieval application takes
  about 30 minutes
• Used in many MetBroker demo applications

33
Easy Development of New Applications
34
Databases Linked Using Drivers
Databases
Applications
Broker
35
Linking a Database to a Broker

• Need to write one Java class (a driver) with
  two or three main methods
• Typically use Java Database Connectivity (JDBC)
  drivers to query relational databases
• Can reuse existing classes for Oracle, Postgres,
  and MySQL JDBC.
• Drivers can read data from files via HTTP, or
  parse HTML tables.
• Test harness available for driver development

36
How much work?

• Add a new database to MetBroker
• roughly working - 3 days
• tested more thoroughly 1 week
• (assuming that database is readily accessible)
• Make a MetBroker applet
• working prototype in one day
• (assuming basic science is done)

37
Developer Resources

• Demos, documentation, downloads, pictures
• http//www.agmodel.net
• Mailing lists subscribe at web site
• Brokers are open source under the GNU Lesser
  General Public License.
• Can be used in commercial applications
• Any improvements must be released under same
  license

38
Reminders

• Linking your data to the grid lets you use grid
  tools developed by others.
• It doesnt mean you must share your data with
  everyone (password protection).
• The key to interoperability is the interface

39
Contents

• Concepts
• Remote Procedure Calls (RPC)
• Brokers
• Using a broker
• Metadata requests
• Data requests
• Issues
• Linking your own data
• SOAP
• New brokers and future enhancements

40
SOAP

• An XML-based RPC mechanism
• Independent of programming language
• SOAP toolkits for C, Visual Basic, Delphi,
  Java, ASP, PHP, Perl, Python
• Firewall-friendly
• Basis of Microsoft .NET architecture
• A SOAP Web service is described in standard
  format using a .WSDL file(Web Services
  Description Language)

41
Example - MetSOAP
Visual Basic client application
VB function calls
Application code
MS SOAP Toolkit
VB objects
SOAP over HTTP
MetBroker
MetSOAP
RMI
Delphi client application
Delphi function calls
SOAP over HTTP
Applicationcode
Borland SOAP Toolkit
Delphi objects
42
SOAP Summary

• Positives
• Opens brokers to a range of programming languages
• Already the standard for Web-based RPC
• Avoids most firewall issues
• Negatives
• Cant reuse Java classes - must mirror in each
  language
• Interoperability is still imperfect (but
  improving fast)
• Inefficient for large data sets

43
Typical Array in SOAP

• ... 
• ltitem xsitype"xsdfloat"gt23.43lt/itemgt
• ltitem xsitype"xsdfloat"gt23.43lt/itemgt
• ltitem xsitype"xsdfloat"gt23.43lt/itemgt
• ltitem xsitype"xsdfloat"gt21.79lt/itemgt
• ltitem xsitype"xsdfloat"gt21.79lt/itemgt
• ltitem xsitype"xsdfloat"gt23.3lt/itemgt
• ltitem xsitype"xsdfloat"gt21.11lt/itemgt
• ltitem xsitype"xsdfloat"gt19.49lt/itemgt
• ltitem xsitype"xsdfloat"gt18.48lt/itemgt
• ltitem xsitype"xsdfloat"gt16.52lt/itemgt
• ltitem xsitype"xsdfloat"gt14.73lt/itemgt
• ltitem xsitype"xsdfloat"gt14.73lt/itemgt
• ...

44
Model Deployment Options - Summary

• Java applet
• flexible, responsive once loaded but requires
  plug-in
• Java servlet
• simple browsers, rapid load, but clunky interface
• SOAP
• direct access for Delphi, VB
• Existing models (FORTAN,C,BASIC)
• Use simple Java app. to retrieve data and write
  to a file

45
Contents

• Concepts
• Brokers
• Remote Procedure Calls (RPC)
• Using a broker
• Metadata requests
• Data requests
• Issues
• Linking your own data
• SOAP
• New brokers and future enhancements

46
New Brokers

• DEM Broker - Digital Elevation Model data
• Japan 50m resolution
• World 1km resolution
• ResourceServer
• Internet-based service for software localization
• ChizuBroker Web map services
• www.mapfan.com (Japan)
• www.mapzone.co.nz (New Zealand)
• SoilBroker soils data

47
Future enhancements

• Broker-side interpolation (using DEMBroker)
• temporal
• spatial

48
Virtual stationBroker-side interpolation at
single site
Client
Server
Temperature
Rainfall
Radiation
49
Future enhancements

• Broker-side interpolation (using DEMBroker)
• temporal
• spatial
• Other brokers
• Place names
• Crops and crop models (prototyped)
• Agrichemicals

50
Final Points

• The Grid provides a stable, powerful foundation
  for Ag and NR software
• You can now develop grid applications in your
  favourite programming language.
• NARO is keen to help you to both develop new
  applications and link new databases.
• Database owners around the world will be keen to
  evaluate your applications.

51
Thank you