(Bregman?????)

1
???????????????

• ?????????????
• ????

2
??????????(?)

• ?????????????.
• ?
• ????????????????
• ?????????????
• (Bregman?????)
• ??????????????
• (?????????????????)

3
??????

• ?????(Structured Convex Optimization)
• ??
• Bregman??,Bregman divergence
• ??????????
• ?????(Proximal Gradient method)
• ???????????
• ????????
• ?????????
• ??
• Bregman???????
• ???????????????
• ???

4
????????????????(structured convex
optimization, composite convex optimization)

• ?????
• h ?????????
• P ?????(??)????
• ? lasso , group lasso
  
• ????

5
???1
L1?????
g ?????
6
???2
???????????? ???, ????? ????????????
?????. ?? ??????????????
Collins, Globerson, Carreras, Bartlett,
2007 CT?????
Ben-Tal, Margalit, Nemirovski, 2001
7
???

• ???????SVM
• ???,

8
??????

• ?????(Structured Convex Optimization)
• ??
• Bregman??,Bregman divergence
• ??????????
• ?????(Proximal Gradient method)
• ???????????
• ????????
• ?????????
• ??
• Bregman???????
• ???????????????
• ???

9
Bregman??(Bregman divergence)

• ??????(??)??????.
• Bregman??
• ??1
• ??2
• ?1 ???
• ?2 ???

????????
10
(???)???????

• ????????? ???.
• ?????????? ???????
• ?? ??????????????.
• (i)
• (ii)
• ?? ??????(?????????)
• ??????
• ???????

11
????????????

• ??????
• Hoffman, 1952
• L1?????
• Tseng, 2010
• ??? ?, ?????????????

12
??????

• ?????(Structured Convex Optimization)
• ??
• Bregman??,Bregman divergence
• ??????????
• ?????(Proximal Gradient method)
• ???????????
• ????????
• ?????????
• ??
• Bregman???????
• ???????????????
• ???

13
????????????
????
Rockarfellar 1976 Censor, Zenios 1992, etc

• ????????? ??????????????????????
• Luque, 1984
• ??????????

????(Mirror Descent?)
Nemirovsky, Yudin, 1983

• ?????
• f ?????????, ??????????,
• ??????????.

14
?????
?????
h????? P????????????????
P??????
h???????,P???????????
15
?????????(1)

• ??1 ??? ????????.
• ???????
• ???L????????? ??? ?????????,
• ???????????.

?????
? ??????????????????????.
16
?????????(2)

• ??2 ??1???????????.
• ?????????????????????.
• ????, ? 0 ???????.
• ????????????????????
• h ?????
• L1?????

17
????????

• ?????? ???????????.
• ????
• ?????????????????.
• Key Bregman?????
• P??????????????

18
???1L1?????

• Bregman??? ?????

????
Iterative Shrinkage Threshold Algorithm (ISTA)
19
???2???????????
????

• ???Bregman???????

Exponentiated Gradient Algorithm (EG)
20
???3???????SVM
????

• ???Bregman???????
• ?????KKT???

h??2??? P????1??? N????-1???
21
???3???????SVM
O(n) ?????
22
?????????

• ???????????????
• ????????????????
• ???????.

23
?????1 Nesterov, 1983, Beck, Teboulle, 2009

• ???

ISTA
Fast ISTA (FISTA) Beck and Teboulle, A fast
iterative shrinkage-thresholding algorithm for
linear inverse problems SIAM Journal on Imaging
Sciences, 2009.
???
24
?????2 Nesterov, 1988, Auslender, Teboulle,
2006
???
25
?????3Nesterov 2005, Tseng, 2008
???
26
?????????
?????
ISTA
P?????
????
????? Mirror Descent ?
????
FISTA
EG?
???EG? X. Zhang, A. Saha, S. V. N. Vishwanathan,
Accelerated Training of Max-Margin Markov
Networks with Kernels, Algorithmic Learning
Theory (ALT), 2011
27
??????

• ?????(Structured Convex Optimization)
• ??
• Bregman??,Bregman divergence
• ??????????
• ?????(Proximal Gradient method)
• ???????????
• ????????
• ?????????
• ??
• Bregman???????
• ???????????????
• ???

28
?????????
???????????,ISTA,FISTA?EG?? P??????
?? h ??????????(???????) ???????

• ??????
• ??? (??????)
• ???????? (???????)
• ???????

29
??1??h ????Bregman?????

• ?? ????,
• ?????????????

Bregman???
???????????????!!
30
Bregman??????

• ???? ????????
• ?1 ????????????
• ?2 ??(????)?????????

D???????
31
??2 ???

• ?????? ??????????????
• ???
• ??????????????
• ? ???-?????
• Coordinate gradient descent ?
• SVM?Sequential Minimal Optimization ?

32
????????? ???

• ?
• ??,
• ?????,????????.( ????????.)
• ?????? ???????,

L1-L2???
?????????.(????????????)
33
?????????? ??????

• ? ??????,?????k????
• ?
• ????

34
???

• ?????????????????
• ??????????.
• ??????????
• Bregman?????
• ?????????????
• ??????????????,???????
• ????????????????(??)

35
(???????)??????a

• ?????
• Paul Tseng, Approximation accuracy, gradient
  methods, and error bound for structured convex
  optimization, Mathematical Programming, Ser, B,
  125, pp. 263-296, 2010.
• ????????
• Jong-Shi Pang, Error bounds in mathematical
  programming, Mathematical Programming , 79, pp.
  299-332, 1997.
• ????
• http//www.optimization-online.org/

36

• ???,???????????