Use of the Teragrid for Subsurface Modeling and Oil Reservoir Management Studies

1
Use of the Teragrid for Sub-surface Modeling and
Oil Reservoir Management Studies

• Benjamin Rutt
• Tahsin Kurc
• Umit Catalyurek
• Joel Saltz
• Biomedical Informatics Department
• The Ohio State University
• June 13, 2006

2
Collaborators
Mary Wheeler, Hector Klie, Wolfgang Bangerth
http//www.ices.utexas.edu/CSM University of
Texas at Austin
Bruce Loftis David McWilliams Ruth Aydt Tim
Cockerill http//www.teragrid.org TeraGrid
Joel Saltz, Benjamin Rutt, Tahsin Kurc, Umit
Catalyurek, Krishnan Sivaramakrishnan, Michael
Zhang Multiscale Computing Lab http//www.multisca
lecomputing.org Biomedical Informatics
Department The Ohio State University
Manish Parashar, Viraj Bhat http//www.caip.rutge
rs.edu/TASSL Rutgers University
Pete Wyckoff, Leslie Southern http//www.osc.edu T
he Ohio Supercomputer Center
Paul Stoffa, Mrinal Sen, Roustam Seifoullaev
Institute for Geophysics University of Texas at
Austin
3
Knowledge-based Data-driven Management of
Subsurface Geosystems
Detect and track changes in data during
production. Invert data for reservoir
properties. Detect and track reservoir
changes. Assimilate data/reservoir properties
into evolving reservoir model. Use simulation and
optimization to guide future production.
4
Vision Diverse Geosystems Similar Solutions
Landfills
Oilfields
Models
Simulation
UndergroundPollution
UnderseaReservoirs
Data
Control
5
Effective Oil Reservoir Management

• What is an Oil Reservoir
• Surface Facilities Man-made objects such as
  productions wells, off-shore platforms, injection
  wells
• Subsurface Multiple geophysical domains
  interacting with each other
• Why is it important
• Better utilization of existing reservoirs
• Discovering new reservoirs
• Minimizing adverse effects to the environment

Better Management
Bad Management
Less Bypassed Oil
Much Bypassed Oil
6
Utilizing Generated and Gathered Data

• Production of oil and gas can take advantage of a
  better understanding of the reservoirs state
• Knowledge of the reservoirs state during
  production can result in better engineering
  decisions
• Economical evaluation
• Physical characteristics (bypassed oil, high
  pressure zones)
• Productions techniques for safe operating
  conditions in complex and difficult areas
• Refine numerical models
• Incorporate seismic data (simulated and/or
  measured)
• History matching

7
Oil Well Placement
8
Production Simulation via Reservoir Modeling
Monitor Production by acquiring Time Lapse
Observations of Seismic Data
Model 1
Model N
Data Analysis
New Model or Parameters
Data Management and Manipulation
Revise Knowledge of Reservoir Model via Imaging
and Inversion of Seismic Data
Modify Production Strategy using an Optimization
Criteria
Data Parameters
9
Computational and Storage Requirements

• Numerical simulations
• Compute intensive large models take too long to
  execute.
• We need many evaluations to completely
  characterize the solution surface, we may have to
  do thousands of computations
• Simulations are executed at multiple sites
• On-the-fly steering of optimization processes are
  desired
• Very large-scale oil and geophysical data
• Thousands of simulations can generate 100TB
  datasets for a single reservoir
• Data stored in files
• Enable on-demand exploration and comparison of
  multiple scenarios
• Integration of multiple data types (reservoir
  simulation, seismic)
• Support for data subsetting, filtering,
  aggregation

10
Software Tools

• Integrated Parallel Accurate Reservoir
  Simulation IPARS
• Multiple individual physical models and
  algorithms for multiphase flow and transport.
• Provides linear solvers with state of the art
  preconditioners.
• Couplings with geomechanics and chemistry
• Multiblock approach (subdomain can treat
  unstructured grids)
• Seismic Data Simulation FDPSV
• Simulation of seismic data gathering
• Simulates sound traces shot from sound sources
  and captured by receivers
• Can scale up to thousands of sources and
  receivers
• Simulation for each source can be run independent
  of that of another source
• Optimization Tools
• Very Fast Simulated Annealing (VFSA)
• Simultaneous Perturbation Stochastic Optimization
  (SPSA)

11
Software Tools

• Grid Computational Collaboratory Discover
• Seamless and secure access to and interactions
  between users, applications, and services
• P2P Grid middleware, autonomic composition,
  collaborative portals
• Data Virtualization STORM
• Large data querying capabilities
• Distributed data virtualization
• Indexing, Data Cluster/Decluster, Parallel Data
  Transfer
• Data Analysis/Processing Workflows DataCutter
• Filter-stream based Framework for Combined
  Task/Data Parallelism
• On demand data product generation

12
Software Tools and Interoperability
Static data
STORM
Visualization
Data manag./ assimilation
Clients
Discover
Steering Monitoring
Dynamic data
Objective function
IPARS
Geophysics
Collaboration
13
Leveraging the TeraGrid Infrastructure

• Compute Power
• Running thousands of simulations and
  optimizations
• Large scale seismic simulation runs corresponding
  to surveys with large numbers of sources and
  receivers
• Storage Power
• Total size of datasets in the range of multiple
  terabytes
• Distributed, high-performance storage systems
• Long term archival storage of data
• On-demand querying and analysis of data
• Knowledge base
• Consultant David McWilliams of NCSA helped us
  generate correct results and reduce simulation
  running time of the seismic simulation code from
  gt24 hours per job to 4-8 hours by suggesting new
  compilation flags

14
Large Scale Data Management and Querying

• Very large-scale oil and geophysical data
• Thousands of simulations can generate 100TB
  datasets for a single reservoir
• Data stored in files
• Simulations are run at multiple sites
• Each cluster has its own batch queue and storage
  heirarchies with unique characteristics
• Common driver script is used to submit jobs
  across all sites with local differences
  abstracted out
• Enables on-demand exploration and comparison of
  multiple scenarios
• Distributed querying support on flat files
• Support for data subsetting, filtering,
  aggregation

15
Datasets (SDSC)
Seismic Sim
Seismic Sim
Seismic Sim
Distributed Execution
Data Manipulation Tools
Datasets (NCSA)
Seismic Sim
Seismic Sim
Seismic Sim
Datasets (UC)
16
Data Virtualization Support STORM

• Support efficient selection of the data of
  interest from distributed scientific datasets and
  transfer of data from storage clusters to compute
  clusters
• Data stored in distributed collections of files.
• Data Virtualization and Subsetting Model Grid
  virtualized object relational database
• Virtual Tables
• Select Queries
• Distributed Arrays

SELECT ltDataElementsgt FROM Dataset-1,
Dataset-2,, Dataset-n WHERE ltExpressiongt AND
ltFilter(ltDataElementgt)gt GROUP-BY-PROCESSOR
ComputeAttribute(ltDataElementgt)
17
Selection of Records of interest using STORM
Seismic Datasets 10-25GB per file. About 30TB
of Data.
18
Conclusion

• Have executed seismic simulations across multiple
  TeraGrid sites, totaling 30TB
• NCSA
• SDSC
• UC
• Have developed a query system that can extract
  knowledge from and visualize these datasets
• Future work includes looking into wrapping the
  simulation and querying activities up in a higher
  level tool with an easy to use interface for
  physical scientists to use

19
Extra slides follow
20
Selection in Seismic Data