Loading...

PPT – Automated Translation of Dynamic Programming Problems to JAVA Code and their Solution via an Interme PowerPoint presentation | free to view - id: 11b04-ZjUyM

The Adobe Flash plugin is needed to view this content

Automated Translation of Dynamic Programming

Problems to JAVA Code and their Solution via an

Intermediate Petri Net Representation

ICS800 Dissertation Research

Dissertation Defense February 28, 2005

Holger Mauch

Motivation and Contributions

- No general DP solver existed previously
- DP2PN2Solver is fairly general
- No general specification language with DP focus

existed - gDPS designed exclusively for DP problems
- Debugging a flawed DP model is difficult
- Intermediate Bellman net representation allows

consistency checks - A novel low-level Bellman net model helps

Overview

- Specification of a DP problem
- Translation to a Petri net (PN) model
- Consistency checks on PN level
- Translation to executable solver code
- JAVA solver code
- Spreadsheet
- PNML code for simulation of PN
- Executable code yields solution to problem

Overview DP2PN modules

Overview PN2Solver modules

Dynamic Programming (DP)

- general optimization technique
- here focus on discrete DP (as opposed to

continuous DP)

Principle of Optimality

- an optimal set of decisions has the property

that whatever the first decision is, the

remaining decisions must be optimal with respect

to the outcome which results from the first

decision (Bellman, Dynamic Programming, 1957)

Definition of DP

- no unique mathematical definition
- ... a DP problem is any optimization problem

that lends itself to a formulation that yields an

equation known as the DP optimality equation.

(Sniedovich 1992) - DP is recursive optimization (Nemhauser 1966)

18 DP Problem Examples...

- Optimal Binary Search Tree (BST)
- Edit Distance Problem (EDP)
- Fishery Profits Problem (FPP)
- Integer Linear Programming (ILP)
- 0/1 Knapsack Problem (KS01)
- Longest Common Subsequence (LCS)
- Longest Simple Path (LSP)
- Matrix Chain Multiplication (MCM)
- Optimal Financing Problem (OFP)

- Optimal Storage on a Tape (OST)
- Reject Allowance Problem (RAP)
- Reliability Design Problem (RDP)
- Shrub Covering Assortment (SCA)
- Stagecoach Problem (SCP)
- Shortest Path Acyclic (SPA)
- Shortest Path Cyclic (SPC)
- Traveling Salesman Problem (TSP)
- Winning in Las Vegas (WLV)

...from these Textbooks

- Cormen et al., Introduction to Algorithms BST,

LCS, LSP, MCM, SCP, SPC - Hillier/Lieberman, Introduction to OR ILP, RAP,

RDP, SCP, WLV - Horowitz et al., Computer Algorithms BST, EDP,

KS01, RDP, SCP, SPA, TSP - Winston, OR Applications and Algorithms FPP,

KS01, OFP, SCP, TSP

Matrix Chain Multiplication (MCM) Problem

- Given product A1A2...An of matrices
- Find optimal parenthesization
- dimensions of Ai are denoted by di-1 and di
- DP functional equation f(i,j)mini?k1,j)di-1dkdj if iopt. parenth. for Ai...Aj
- Base f(i,j)0 if ij
- Goal Find f(1,n)

MCM Problem Instance

- n4
- (d0,d1,d2,d3,d4)(3,4,5,2,2)
- f(1,4)76 elementary multiplications in
- (A1(A2A3))A4 optimal parenthesization

DP Technology

- No standardized specifications (as in linear

programming (LP)) existed before gDPS - No general DP solver software existed before

DP2PN2Solver - needed specialized software for each problem
- develop specialized software application or
- develop specialized spreadsheet template
- OR/MS people often lack software development

skills

Improvement

- Define the structure of a DP problem in a

standardized specification - Many DP problems of practical relevance have

similar structure - Compare to LP problem specification opt

zc1x1...cnxn s.t. constraints

ai1x1...ainxnbi - DP specification will be more complex and not

cover all DP problems

DP2PN2Solver Steps

- Step (1) requires human intelligence, all other

steps are automated - Human modeler design the DP problem

specification for the real-world problem and

collect the input data for the problem instance

to be solved - DP2PN module produces Bellman net
- PN2Solver module produces executable code
- Run the resulting executable solver code to get

the solution to the problem instance

Elements of DP Specification

- Multistage decision model
- States (Stage is part of a state)
- Decision (Actions)
- Rewards (Stage return, costs)
- Transformation (Transition function)
- Goal state (Goal of computation)
- DP functional equation
- Recursive relationship
- Base case(s) (Base states their function values)

DP Elements of MCM

- State s(i,j) (a pair of integers)
- Decision k?D(s)i,..,j-1
- Reward r(s,k)di-1dkdj
- Transformations
- t1(s,k)(i,k)
- t2(s,k)(k1,j)
- Goal state (1,n). Goal of computation f(1,n)
- DP functional equation
- Recursive relationship f(s)mink?Df(t1(s,k))f(t2

(s,k))r(s,k) - Base cases f(i,j)0 if ij (base states (i,j) if

ij)

MCM example in general DP Specification Language

(gDPS)

- BEGIN
- NAME MCM //MatrixMultiplication
- GENERAL_VARIABLES_BEGIN
- //dimensions in MatMult problem
- private static int dimension 3, 4, 5, 2,

2 - private static int n dimension.length-1

//n4 matrices - GENERAL_VARIABLES_END
- //a state is a vector/list of primitive or Set

types - STATE_TYPE (int i, int j)
- DECISION_VARIABLE int k
- //the set of alternatives is a function of the

state - DECISION_SPACE D(i,j)i,..,j-1

MCM example in gDPS (contd.)

- GOAL f(1,n)
- DPFE_BASE_CONDITIONS
- f(i,j)0.0 WHEN (ij)
- DPFE f(i,j)
- MIN_k IN D
- f(t1(i,j,k))f(t2(i,j,k))r(i,j,k)

- REWARD_FUNCTION r(i,j,k)dimensioni-1
- dimensionkdimensionj
- TRANSFORMATION_FUNCTION t1(i,j,k)(i,k)
- t2(i,j,k)(k1,j)
- END

Elements of gDPS

- GENERAL_VARIABLES section (optional)
- int, double, String, etc. in C/Java style

syntax - SET_VARIABLES section (optional)
- Set variables in mathematical notation
- GENERAL_FUNCTIONS section (optional)
- arbitrary functions in C/Java style syntax
- STATE_TYPE section (mandatory)
- vector of state variables of heterogeneos types
- DECISION_VARIABLE and DECISION_SPACE sections

(mandatory)

Elements of gDPS (contd.)

- DPFE and DPFE_BASE_CONDITIONS section (mandatory)
- GOAL section (mandatory)
- declare goal of the computation (start state)
- REWARD_FUNCTION section (mandatory)
- TRANSFORMATION_FUNCTION section (mandatory)
- define one or more
- TRANSITION_WEIGHTS section (optional)
- for discounted or probabilistic DP problems

Petri Net Model for DP Instances

- Observation The main operators for solving DP

problem instances are - Minimization/Maximization
- Addition (rewards add up) and multiplication
- These operators can easily be modeled with Petri

nets - Idea Transform gDPS specification into a PN

model with the specification parser DP2PN

Petri Net Model for DP Instances

- Weighted directed graph not sufficient
- Would like exactly one edge per decision, but one

decision might lead to multiple successor states - Tree model would not take advantage of

overlapping subproblems

Place-Transition Nets

Definition of Place-Transition Net

- A 6-tuple (P,T,F,K,M0,W) is called a P/T-net, iff
- (P,T,F) is a finite net
- K P???? gives a capacity for each place
- W F??-0 attaches a weight to each arc of the

net - M0 P ???? is the initial marking, respecting

the capacities, i.e. M0(p)?K(p) ?p?P.

Activation of a Transition

- A transition t is activated if
- for every place p from the preset t the number

of tokens at p is greater or equal to the weight

W(p,t) of the arrow from p to t - for every place p in the postset t the number of

tokens at p plus the weight W(t,p) of the arrow

from t to p does not exceed the capacity K(p) of

p.

Example Activated Transition

Firing of a transition

- An activated transition t may fire by
- decreasing the number of tokens ?p?t by W(p,t)
- increasing the number of tokens ?p?t by W(t,p).

Example Firing

Petri Net Models for DP

- 2 suitable PN models (Bellman nets)
- Low-level Bellman nets (LLBN) use

Place/Transition net semantics - High-level Bellman nets (HLBN)
- based on the emerging PNML standard (Billington

et al. 2003) - closely related to Colored Petri Nets (Jensen

1992) and Predicate/Transition nets (Genrich

1987)

Low-Level Bellman Net Model

Low-Level Bellman Net Model

High-Level Bellman Net Model

High-Level Bellman Net Model

Low-Level Bellman Net

- Mauch 2004
- Places
- state place for each state
- min-in place for each decision
- Transitions
- min-transition for each minimization operation
- copy transition transferring tokens from a state

place to the appropriate min-in places

High-Level Bellman Net

- Lew 2002, Lew/Mauch 2003, Lew/Mauch 2004
- real-valued tokens, at most one per place
- Transitions
- M-Transitions (for min/max)
- E-Transitions (for addition, multiplication)
- Places
- state places for each state
- intermediate places
- enabling places (technicality that prevents

transitions from firing more than once)

LLBN vs. HLBN (part 1/2)

- LLBN
- theoretical analysis easier
- optimal policy can be determined immediately from

final marking - no synchronization necessary, transitions can

fire concurrently, any order - HLBN
- more expressive than LLBN, easier to express

complex functionality - more efficient transformation to solver code
- some synchronization necessary, limited degree of

concurrency

LLBN vs. HLBN (part 2/2)

- LLBNs and HLBNs shown to be equivalent for a

subclass Pnat,min, of DP problems where - functional values are natural numbers and
- direction of optimization is minimization and
- operator connecting functionals in DPFE is

addition - Proof is constructive for Pnat,min, conversion

algorithms have been designed - Existing PN software packages (e.g. Renew) can

simulate both LLBN and HLBN - DP2PN2Solver uses HLBN

DP2PN Software Module

- DP2PN parser created with JavaCC

(compiler-compiler) - integrates seamlessly into Java environment
- Catches syntactical errors like
- No goal state provided
- No or illegal DP functional equation provided
- No base case for the DP functional equation

provided. - No decision variable provided
- No decision set provided

DP2PN Software Module

- State-Decision-Reward-Transformation-Table

produced by algorithm - initialize the set statesSeen to contain the goal

state - initialize the queue stateQueue to contain the

goal state - while stateQueue not empty
- currentStatedequeue state from stateQueue
- if currentState is not a base state
- loop over all possible decisions
- generate table entry (a row for the
- state-decision-reward-transformation-table)

- enqueue those next states which are not in

statesSeen - into the stateQueue and also add them to

statesSeen

State-Decision-Reward-Transformation Table for MCM

Output of DP2PN (File Format)

- view MCMBN.csv in MS-Excel

Consistency Checks on PN Level

- Challenge Certain DP specification design

mistakes are difficult to detect. - Examination of Bellman net might help.
- PN theory developed algorithms to check
- structural (static) properties
- circularity (e.g. SPA DP model used for SPC)
- initially unmarked source place (e.g. a base case

has been forgotten) - no sink place representing the goal state place

(e.g. mistake in transformation function) - behavioral (dynamic) properties
- premature deadlock (e.g. SPA DP model used for

SPC) - liveness (e.g. invalid/omitted recursion base)
- Algorithms verify the semantics of the DP problem

specification - Syntactic validity can be checked by the DP2PN

parser - DPBNs represent DP problem correctly

PN2Solver From PN to Executable

- Input PN representation of DP problem instance
- Output solver code, which solves the DP problem

instance - Three PN2Solver software modules
- PN2Java
- PN2XML
- PN2Spreadsheet

PN2Java From PN to Java

- User is not dependent on PN simulators, but can

use ubiquitous Java platform - Automatically produced Java solver code
- is documented with comments
- can be manually refined, if desired
- can be used as building block for larger, problem

specific software development project

PN2Java Output (part of MCM)

- List argumentList //reusable temporary

variable used for min/max - List operandList //reusable temporary

variable for eval. (Etransitions) - //code to initialize base state place

variables - CalculationObject I5 new CalculationObject("

(1,1)",0.0) - CalculationObject I20 new

CalculationObject("(4,4)",0.0) - CalculationObject I25 new

CalculationObject("(2,2)",0.0) - CalculationObject I36 new

CalculationObject("(3,3)",0.0) - //code that simulates the transitions firing
- operandListnew ArrayList()
- operandList.add(I5)
- operandList.add(I25)
- ECalculationObject I32 new

ECalculationObject( - "k1",
- I5.getValue()I25.getValue()60.0,
- operandList)
- argumentListnew ArrayList()
- argumentList.add(I32)

Java Solver Output (MCM example)

- The optimal value is 76.0
- The solution tree is
- State (1,4) has optimal value 76.0
- Decision k3
- State (1,3) has optimal value 64.0
- Decision k1
- Base state (1,1) has initial value 0.0
- State (2,3) has optimal value 40.0
- Decision k2
- Base state (2,2) has initial value 0.0
- Base state (3,3) has initial value 0.0
- Base state (4,4) has initial value 0.0

- optimal parenthesization (A1(A2A3))A4 ((A1)((A2)

(A3)))(A4)

PN2XML From PN to PNML

- PNML is XML-based interchange format for PN
- standardization of PNML in progress
- Import PNML file into PN software systems
- Simulate the Bellman net
- Graphical visualization of solution process
- PN simulation software often allows the automated

analysis of certain PN properties helps with

consistency checks - Overhead to run PN software
- Reconstruction of opt. policy not automated
- Do demonstration with Renew software system

PN2Spreadsheet

- Popularity of spreadsheet, especially in OR/MS

community - Reconstruction of optimal policy not automated
- link to MCM example

Future Work

- Add a graphical front-end as a more convenient

user interface - Extend LLBN model (e.g. for multiplicative DP

functional equations) - Analysis of Bellman nets
- automate consistency checks
- consult PN theory to find more properties that

can be examined - Potential for parallelization. Concurrency

property in Bellman net identifies computations

that can be performed in parallel.

Summary of Contributions (part 1)

- Designed and defined gDPS source language
- general
- easy to learn (especially for nonprogrammers)
- Modeled numerous DP applications
- illustrates usefulness as educational tool
- Theoretically defined LLBN, HLBN, DPBN
- conform to standard conventions
- permit analyses
- Analyzed Bellman nets and their theoretical

properties

Summary of Contributions (part 2)

- Developed internal representation for Bellman

nets. Flexibility allows - different sources
- different execution systems
- Implemented
- preprocessor for ILP
- compiler from gDPS to BN
- solver code generators
- from BN to Java
- from BN to Spreadsheet
- from BN to PNML (permits use of PN tool as

analyzer and execution system) - Designed DPBN error analysis routines

Summary of Contributions (part 3)

- Designed overall system structure that
- is general, flexible, user-friendly
- is adaptable to needs/preferences of users
- in educational
- and certain commercial settings
- allows automated analyses
- can serve as framework for a more advanced system