View source for Bayesian Ying Yang learning

==Ying-Yang best harmony : Ying-Yang best matching in a least complexity ==

We proceed to understand of the theory of Ying-Yang best harmony. We start to refer the  modern perspective on the Chinese ancient Yin-Yang theory introduced in the Appendix of  [[#BYY10|one recent overview]] (Xu, 2010) and then elaborated  in    Sect. 3.1 of  [[#BYY12|another  overview]] (Xu, 2012).  As illustrated in <figref>byy-yy.gif</figref>3,  a body or a system that survives and interacts with  its world can be regarded as  a Ying-Yang system that functionally composes of two complement parts. One is called Yang,  from its external world (called Yang domain  <math>X</math>) via one  pathway  (called Yang pathway  <math> X \to R </math>) that transforms what gathered in the Yang domain into inner representations in  its inside  domain (called Ying domain  <math> R </math>);  while the other is called Ying that consists of both the Ying domain  <math>R</math> that receives, accumulates, integrates, digests the inner representations provided by Yang, and a Yang pathway  <math> R \to X </math> that selects among the Ying domain to generate outputs to its external world.

[[Image:byy-yy.gif|thumb|500px|right|figyy| BYY system from a perspective on Yin-Yang]]

Referring to <figref>byy-yy.gif</figref> 3(b) & (c) & (d),  Ying is primary, featured with a compact capacity (i.e., in a least complexity) of accommodating and accumulating, as well as a good ability of integrating and digesting whatever supplied by Yang; while Yang is secondary, fetching from the outside and serving the demands of Ying, and thus is vigor in adapting to variety of not only external world but also the dynamic range of the Ying. A Ying-Yang system inputs by Yang and outputs by Ying, operating in a Wu-Xing circling flow during which  Ying and Yang adapt each other to cooperate under a best harmony principle, see <figref>byy-yy.gif</figref> 3(e) & (f).  In the probabilistic framework, we get the ''Bayesian Ying-Yang (BYY) system'' by eq.\eqref{two},  in which the Ying uses <math> q({X} )= \int q({X} | {R})q({R}) d{R} </math> to fit observed samples <math>\{x_t,  t=1, \cdots, N\}</math> with  help of <math> p({R}|{X})</math>, not in a maximum likelihood sense but maximizing the harmony measure by eq.\eqref{hmbyy}, such that all the unknowns  in this BYY system are determined. This maximization leads to not only a Ying-Yang best match but also a BYY system in a least complexity, which makes both   '''parameter learning''' and  '''model selection'''  by the same  measure  <math>H(p \Vert q)</math>.

This nature can be further examined  from the following three information theoretic perspectives:

* Considering the following relation
:<math>\label{h-kl}
\begin{array}{l}
H(p \Vert q) = H(p \Vert p) - KL(p \Vert q), \\
 H(p \Vert q) =\int p(u)\ln{q(u)}du, \  KL(p \Vert q)= \int p(u)\ln{ p(u) \over q(u)}du,
\end{array}
</math>
where <math> KL(p \Vert q)</math> is  Kullback-Leibler divergence while  <math>- H(p \Vert p) </math> is entropy, we observe that  maximizing <math> H(p \Vert q) </math>  by eq.\eqref{hmbyy} consists of not only minimizing <math> KL(p({R}|{X})p({ X} )\Vert q({X} | {R})q({R}))</math> for a Ying-Yang best match but also minimizing the information <math>- H(p({R}|{X})p({ X})\Vert p({R}|{X})p({ X})) </math> of Yang, and thus in turn  the information of Ying,  due to the Ying-Yang best match nature. That is, ''Ying and Yang seeks a best agreement in a most tacit manner featured with a least amount of information communication in its operation flow''.

* For a fixed <math> p</math>,  maximizing <math> H(p \Vert q) </math> with respect to a free <math> q</math> will force  <math> q=p</math>, which forces the Ying <math> q({X} | {R})q({R})</math> to best  match  the Yang <math> p({R}|{X})p({X})</math>.
Due to a finite size <math>N </math> and other existing constraints (if any), the limit <math>q({X} | {R})q({R})= </math>  <math>p({R}|{X})p({ X} )</math> may not be really reached. Still there is a trend towards this equality by which <math> H(p \Vert q) </math> becomes the negative entropy, and thus  its further maximization will minimize the complexity of this  Ying-Yang system.

* From  the following relation
:<math>\label{h-qy}
\begin{array}{l}
H(p \Vert q) = \int p({R}|{X})p({ X} ) \ln{ q({X} | {R})} dR dX + H(p({R}) \Vert q({R})), \\
 with \  p({R}) = \int p({R}|{X})p({ X} ) dX,
\end{array}
</math>
we may observe that  maximizing <math> H(p \Vert q) </math>  by eq.\eqref{hmbyy} consists of not only maximizing  the  fit of observed samples <math>\{x_t,  t=1, \cdots, N\}</math> by  <math> q({X} | {R})</math> with the  help of <math> p({R}|{X})</math>, but also maximizing <math> H(p({R}) \Vert q({R}))</math> that tends to <math> q({R}) =p({R})</math> and thus approximately <math> H(q({R}) \Vert q({R}))</math>. Therefore, its  further maximization  will minimize the complexity of the inner representation by <math> q({R})</math>. Even without considering priors, i.e., ignoring <math> q(\theta_k) q(k)</math> in eq.\eqref{twospec}, we still have  <math> H(q({R}) \Vert q({R}))=</math>  <math>  H(q(Y|\theta_k^{ y}) \Vert q(Y|\theta_k^{ y}))</math> to be maximized for a least  complexity of  <math> q(Y|\theta_k^{ y})</math>, which actually enforces model selection. Moreover, with help of an appropriate priori  <math> q(\theta_k)</math>, model selection ability will be further improved via the maximization of <math> H(q(Y|\theta_k^{ y}) q(\theta_k)  \Vert q(Y|\theta_k^{ y}) q(\theta_k))</math>.

Also, we may  get further insights on the nature of automatic model selection from observing  the gradient flow <math>\nabla_{ \theta} H(p \Vert q )</math>  for the maximization of <math> H(p \Vert q )</math>,  in comparison with  its counterpart for maximizing the likelihood. For simplicity but without losing generality,  we  ignore a priori  <math>q(\theta )</math> and maximize <math> H(p \Vert q )</math> by eq.\eqref{hmbyy}, resulting in
:<math>\label{hmbyy1}
\begin{array}{l}
\max_{\theta, k} \  H(\theta, k), \  H(\theta, k) = \int p( {Y} | {X}, \theta) p({X})  \ln{[ q({X} | {Y}, \theta^{x|y})q({Y}|\theta^{y})]} d{X} d{Y}.
\end{array}
</math>
As to be addressed in the next subsection, we further design
:<math>\label{pyx}
\begin{array}{l}
p( {Y} | {X}, \theta)= q({X} | {Y}, \theta^{x|y})q({Y}|\theta^{y})/ q({X} | \theta), \  q({X} | \theta)=\int q({X} | {Y}, \theta^{x|y})q({Y}|\theta^{y})d{Y}.
\end{array}
</math>
Then, we  obtain the following gradient
: :<math>\label{flow}
\begin{array}{l}
\nabla_{\theta} H(\theta, k)= \int p( {Y} | {X}, \theta) [1+\Delta_L(X,Y)]\nabla_{\theta}L(X,Y) p({X}) d{X} d{Y},   \\
\Delta_L(X,Y)= L(X,Y,\theta)- \int p( {Y} | {X}, \theta^{y|x}) L(X,Y,\theta) d{Y}, \\ L(X,Y,\theta)=\ln{[ q({X} | {Y}, \theta^{x|y})q({Y}|\theta^{y})]}.
\end{array}
</math>
Noticing that <math> L(X,Y,\theta)</math> describes the fitness of an inner representation <math>Y</math> on the observation <math>X</math>,  <math>\Delta_L(X,Y)</math>  indicates whether the Ying with the currently considered <math> Y </math> fits <math> X </math> better than the average of ones with all the possible choices of <math>Y</math>.

Thus, we get observations similar to what discussed after eq.\eqref{ying-M}. Letting <math>\Delta_L(X,Y)=0</math>, <math> \nabla_{\theta} H(\theta, k) </math> is simplified into <math>\int p( {Y} | {X}, \theta^{y|x})\nabla_{\theta}L(X,Y) p({X}) d{X} d{Y}, \ </math> which is  also be derived from <math>\nabla_{\theta} \int p({X}) \ln{ q({X} | \theta) } d{X} </math>  with <math>q({X} | \theta)</math> by eq.\eqref{pyx}. In other words, with <math>p({ X}  ) </math> at <math>h=0</math>, updating <math>\theta</math> along this  gradient flow  actually implements  the classic ML learning.  With <math>\Delta_L(X,Y)\ne 0</math>,  the gradient flow <math>\nabla_{\theta}L(X,Y)</math> with  all possible choices of <math>Y</math>  is integrated via weighting not just by <math> p({Y}|{X}, \theta^{y|x})</math>  but also by a modification of a relative fitness measure <math>1+\Delta_L(X,Y)</math>. If <math>\Delta_L(X,Y)> 0</math>, updating goes along the same direction of the ML learning but with an increased strength. If  <math>0>\Delta_L(X,Y)> -1</math>,  i.e., the fitness is worse than the average and thus this   <math>Y</math>  is doubtful, updating  still goes along the same direction of the ML  learning, but with a reduced strength. When <math>\Delta_L(X,Y)< -1</math>, updating reverses to the opposite direction, i.e., becoming  de-learning. That is, we again observe that the BYY harmony learning has a RPCL-like nature  and thus gets   a similar  nature of  automatic model selection.