[<< | Prev | Index | Next | >>]

Thursday, August 23, 2018

PDM Unchained

[Research log: breaking the PDM dimension barrier.]

The last term of equation 1 from my prior PDM post can be written in terms of a normal matrix product and the trace* operator (using the shorthand `[]_omega` for `del_omega`, and recalling that `dx//dz` means the Jacobian matrix* of the function mapping `z` to `x`):

` (: del_omega dz/dx , {: dx/dz :}^T :)_F = "tr"(dx/dz [dz/dx]_omega)`

(1)

In that form it's easier to explore what happens if we break our generative function `z->x` into two steps, `z->y->x`, with a hidden layer `y`:

` {: ("tr"(dx/dz [dz/dx]_omega),= "tr"(dx/dy dy/dz [dz/dy dy/dx]_omega)), (,="tr"(dx/dy dy/dz {[dz/dy]_omega dy/dx + dz/dy [dy/dx]_omega})), (,="tr"(dx/dy dy/dz [dz/dy]_omega dy/dx) + "tr"(dx/dy dy/dz dz/dy [dy/dx]_omega)), (,="tr"([dy/dx dx/dy] dy/dz [dz/dy]_omega) + "tr"(dx/dy [dy/dz dz/dy] [dy/dx]_omega)) :}`

(2)

Now if `y` is the same dimensionality as `x` and `z`, and both of our mapping functions are fully invertible, then the Jacobians of the inverse functions are the inverses of the Jacobians, and so they cancel out and we get:

` "tr"(dx/dz [dz/dx]_omega) = "tr"(dy/dz [dz/dy]_omega) + "tr"(dx/dy [dy/dx]_omega)`

(3)

Thus reaffirming that we can chain functions through simple summation of their independent gradients.

However, nothing in equation (2) actually relies on `z->y` or `y->x` being invertible, only that the composite function `z->x` is. For instance, `y` can be of a higher dimensionality than `x` and `z`, in which case all of our Jacobians become rectangular. Equation (2) still holds (but not equation (3)), with the would-be canceling terms acting instead to project the gradient into the subspace of `y` reachable by the auxiliary variable. I.e., the gradient for the `y->x` mapping is conditioned to lie within the subspace of `y` tended by `z`, and so with `z->y` by `x`.

Various implications to ponder.

---

Update 2018-09-04:

A significant constraint in the above is that the joint inverse mapping needs to be valid. For example, for a wider `y` layer, the `z->y` mapping needs to project to the same subspace as the `x->y` mapping, otherwise even though `z->y->z` and `x->y->x` may both be identity, `x->y->z->y->x` may not be! Specifically, be sure that:

` dx/dz = dx/dy dy/dz`

(4)

A consequence of this, somewhat counter-intuitively, is that `dy//dz` needs to track the `x -> y` mapping. Or, alternatively, we can create a distinct `y'` for the generative direction, such that equation (2) becomes:

` {: ("tr"(dx/dz [dz/dx]_omega),= "tr"(dx/(dy') (dy')/dz [dz/dy dy/dx]_omega)), (,="tr"(dx/(dy') (dy')/dz [dz/dy]_omega dy/dx) + "tr"(dx/(dy') (dy')/dz dz/dy [dy/dx]_omega)) :}`

(5)

Again, taking care to insure that the round-trip mapping `x->x` is identity.

[<< | Prev | Index | Next | >>]