2DGS梯度推导

3 minute read

Published:

https://en.wikipedia.org/wiki/Matrix_calculus#Scalar-by-vector_identities

renderCUDA

1. Input:

  • $xy_j$: gaussian point pixel coord
  • pixf: current pix coord
  • $d_j = xy_j - pixf$
  • ${con_o}_j$: [conic_x, conic_y, conic_z, opacity]
    • $[\Sigma^{-1},o]$
    • 这是 2d 的 cov, 3d 的会加粗处理

2. Forward: near to far

  1. $power_j = -0.5*d^T\Sigma^{-1}d$
  2. $G_j = \exp(power_j)$
    1. $\alpha_j = o_j * \exp(power_j) = o_j * G_j$
  4. $ch(annel) += sh_j * \alpha_j *T_j$ \(ch = \sum_{j=0}c_j\alpha_jT_j(\alpha)\)
  5. $T_{j+1} = T_j*(1-\alpha_j)$

3. Backward: far to near

3.1. Original Alpha: $\alpha_j = o_j * \exp(power_j) = o_j * G_j$

  1. dch_dalpha $\frac{\partial ch}{\partial \alpha_j}$

    \[\begin{equation} \begin{aligned} \frac{\partial ch}{\partial \alpha_j} &= \frac{\partial ch}{\partial \alpha_j}\frac{\partial \alpha_j}{\partial \alpha_j} + \sum_{k = j+1}\frac{\partial ch}{\partial T_k}\frac{\partial T_k}{\partial \alpha_j}\\ &= sh_j * T_j + \sum_{k=j+1}sh_k\alpha_k*\frac{T_k}{(1-\alpha_j)}(-1)\\ &= (sh_j -\sum_{k=j+1}sh_k\alpha_k*\frac{T_k}{T_j(1-\alpha_j)})*T_j\\ &= (sh_j -\sum_{k=j+1}sh_k\alpha_k*\frac{T_k}{T_{j+1}})*T_j\\ &= \left(sh_j -\sum_{k=j+1}sh_k\alpha_k\prod^{k-1}_{t=j+1}\left(1-\alpha_{t}\right)\right)*T_j\\ &= \left(sh_j -accum\_rec_{j}\right)*T_j \end{aligned} \end{equation}\] \[\begin{equation} \begin{aligned} accum\_rec_{j} &= \sum_{k=j+1}sh_k\alpha_k\prod^{k-1}_{t=j+1}\left(1-\alpha_{t}\right) \\ &= sh_{j+1}\alpha_{j+1} + sh_{j+2}\alpha_{j+2}(1-\alpha_{j+1}) + sh_{j+3}\alpha_{j+3}(1-\alpha_{j+1})(1-\alpha_{j+2}) + ...\\ &= \alpha_{j+1}sh_{j+1} + (1-\alpha_{j+1})*accum\_rec_{j+1} \end{aligned} \end{equation}\]

  2. dL_dalpha \(\begin{equation} \frac{\partial L}{\partial \alpha_j} = \frac{\partial L}{\partial ch}\frac{\partial ch}{\partial \alpha_j}\\= \frac{\partial L}{\partial ch} *(sh_j - accum\_rec_j) * T_j + （background \ color \ part） \end{equation}\)

    2.1. If $\alpha_j = \left(1-\exp \left(-o_i\mathcal{G}_i \delta_i\right)\right)$

  3. dL_dconic2D
    \(\begin{equation} \begin{aligned} \frac{\partial L}{\partial \Sigma^{-1}} &= \frac{\partial L}{\partial ch}\frac{\partial ch}{\partial \alpha_j}\frac{\partial \alpha_j}{\partial G_j}\frac{\partial G_j}{\partial \Sigma^{-1}} = \frac{\partial L}{\partial G_j}\frac{\partial G_j}{\partial \Sigma^{-1}} \end{aligned} \end{equation}\)

    \[\begin{equation} \begin{aligned} \frac{\partial G_j}{\partial \Sigma^{-1}} &= \frac{\partial G_j}{\partial power_j}\frac{\partial power_j}{\partial \Sigma^{-1}} \\ &= G_j\frac{\partial power_j}{\partial \Sigma^{-1}} = G_j(-0.5*dd^T) \end{aligned} \end{equation}\]

  4. dL_do \(\frac{\partial L}{\partial o_j} = \frac{\partial L}{\partial \alpha_j}\frac{\partial \alpha_j}{\partial o_j} = \frac{\partial L}{\partial \alpha_j} G\)

3.2. Volume Alpha: $\alpha_j = 1- \exp(-o_j * G_j * \delta_j)$

  1. dL_ddelta \(\frac{\partial L}{\partial \delta_j} = \frac{\partial L}{\partial \alpha_j}\frac{\partial \alpha_j}{\partial \delta_j} = \frac{\partial L}{\partial \alpha_j} \exp(-o_j * G_j * \delta_j) * o_j * G_j\) 按照NeRF来说，这个梯度不需要传导，因为采样间隔应该是均匀的。
  2. dL_dG \(\frac{\partial L}{\partial G_j } = \frac{\partial L}{\partial \alpha_j}\frac{\partial \alpha_j}{\partial G_j } = \frac{\partial L}{\partial \alpha_j} \exp(-o_j * G_j * \delta_j) * o_j * \delta_j\)
  3. dL_do \(\frac{\partial L}{\partial o_j} = \frac{\partial L}{\partial \alpha_j}\frac{\partial \alpha_j}{\partial o_j} = \frac{\partial L}{\partial \alpha_j} \exp(-o_j * G_j * \delta_j) *G_j * \delta_j\)

Ray-splat points

\[\mathbf{x}=(x z, y z, z, 1)^{\mathrm{T}}=\mathbf{W} P(u, v)=\mathbf{W H}(u, v, 1,1)^{\mathrm{T}}\] \[\hat{\mathbf{x}}=(x, y, z)^{\mathrm{T}}=(\frac{1}{z}, \frac{1}{z}, 1)\mathbf{W} P(u, v)\]

Depth

Mean depth

\[D=\sum_i \omega_i z_i\]

where

\[\omega_i=T_i \alpha_i \hat{\mathcal{G}}_i(\mathbf{u}(\mathbf{x}))\]

is the weight contribution of the

$i$

-th Gaussian and

\[T_i=\prod_{j=1}^{i-1}\left(1-\alpha_j \hat{\mathcal{G}}_j(\mathbf{u}(\mathbf{x}))\right)\]

measures its visibility.

  • Relative variables
    • \[z_i\]
    • \[\omega_i\]

Calculate

\[z_i\] \[\frac{\partial z_{\text {mean }}}{\partial z_i}=\omega_i\] \[\frac{\partial L}{\partial z_i}=\frac{\partial L}{D}\frac{D}{\partial z_i}=\frac{\partial L}{D}\omega_i\]

Calculate

\[\omega_i\] \[\frac{\partial z_{\text {mean }}}{\partial \omega_i}= z_i\] \[\frac{\partial L}{\partial \omega_i}=\frac{\partial L}{D}\frac{D}{\partial \omega_i}=\frac{\partial L}{D}z_i\]

Calculate

\[\alpha_i\] \[\frac{\partial T_i}{\partial \alpha_i} = T_{i-1} \left(-\hat{\mathcal{G}}_{i-1}(\mathbf{u}(\mathbf{x}))\right)\] \[\frac{\partial \omega_i}{\partial \alpha_i}= T_i \hat{\mathcal{G}}_i(\mathbf{u}(\mathbf{x})) + \alpha_i \hat{\mathcal{G}}_i(\mathbf{u}(\mathbf{x})) \frac{\partial T_i}{\partial \alpha_i}\\ = T_i \hat{\mathcal{G}}_i(\mathbf{u}(\mathbf{x})) - \alpha_i T_{i-1} \hat{\mathcal{G}}_{i-1}(\mathbf{u}(\mathbf{x})) \hat{\mathcal{G}}_i(\mathbf{u}(\mathbf{x}))\]

Distortion loss

\[\begin{aligned} &\begin{aligned} \mathcal{Dist} & =\sum_{i=0}^{N-1} \sum_{j=0}^{i-1} \omega_i \omega_j\left(m_i-m_j\right)^2 \\ & =\sum_{i=0}^{N-1} \omega_i\left(m_i^2 \sum_{j=0}^{i-1} \omega_j+\sum_{j=0}^{i-1} \omega_j m_j^2-2 m_i \sum_{j=0}^{i-1} \omega_j m_j\right) \\ & =\sum_{i=0}^{N-1} \omega_i\left(m_i^2 A_{i-1}+D_{i-1}^2-2 m_i D_{i-1}\right) \end{aligned}\\ &\text { where } A_i=\sum_{j=0}^{i} \omega_j, D_i=\sum_{j=0}^{i} \omega_j m_j \text { and } D_i^2=\sum_{j=0}^{i} \omega_j m_j^2 \text {. } \end{aligned}\] \[A_i=\sum_{j=0}^{i} \omega_j =\sum_{j=0}^{i} \alpha_j \hat{\mathcal{G}}_j(\mathbf{u}(\mathbf{x})) T_j = 1.0 - T_i = 1.0 - \prod_{j=1}^{i}\left(1-\alpha_j \hat{\mathcal{G}}_j(\mathbf{u}(\mathbf{x}))\right)\] \[\frac{\partial \mathcal{Dist}}{\partial m_i}=\omega_i * 2 * m_i * A_{i-1} - 2 D_{i-1} = 2 (\omega_i * m_i * A_{i-1} - D_{i-1})\] \[\frac{\partial \mathcal{Dist}}{\partial \omega_i}=m_i^2 A_{i-1}+D_{i-1}^2-2 m_i D_{i-1}\] \[\frac{\partial \mathcal{Dist}}{\partial \alpha_i}=\frac{\partial L}{\partial \omega_i} \frac{\partial \omega_i}{\partial \alpha_i}\]