https://en.wikipedia.org/wiki/Matrix_calculus#Scalar-by-vector_identities
renderCUDA
1. Input:
- $xy_j$: gaussian point pixel coord
- pixf: current pix coord
- $d_j = xy_j - pixf$
- ${con_o}_j$: [conic_x, conic_y, conic_z, opacity]
- $[\Sigma^{-1},o]$
- 这是 2d 的 cov, 3d 的会加粗处理
2. Forward: near to far
- $power_j = -0.5*d^T\Sigma^{-1}d$
- $G_j = \exp(power_j)$
-
- $\alpha_j = o_j * \exp(power_j) = o_j * G_j$
- $ch(annel) += sh_j * \alpha_j *T_j$
\(ch = \sum_{j=0}c_j\alpha_jT_j(\alpha)\)
- $T_{j+1} = T_j*(1-\alpha_j)$
3. Backward: far to near
3.1. Original Alpha: $\alpha_j = o_j * \exp(power_j) = o_j * G_j$
-
dch_dalpha $\frac{\partial ch}{\partial \alpha_j}$
\[\begin{equation}
\begin{aligned}
\frac{\partial ch}{\partial \alpha_j} &= \frac{\partial ch}{\partial \alpha_j}\frac{\partial \alpha_j}{\partial \alpha_j} + \sum_{k = j+1}\frac{\partial ch}{\partial T_k}\frac{\partial T_k}{\partial \alpha_j}\\
&= sh_j * T_j
+ \sum_{k=j+1}sh_k\alpha_k*\frac{T_k}{(1-\alpha_j)}(-1)\\
&= (sh_j
-\sum_{k=j+1}sh_k\alpha_k*\frac{T_k}{T_j(1-\alpha_j)})*T_j\\
&= (sh_j
-\sum_{k=j+1}sh_k\alpha_k*\frac{T_k}{T_{j+1}})*T_j\\
&= \left(sh_j
-\sum_{k=j+1}sh_k\alpha_k\prod^{k-1}_{t=j+1}\left(1-\alpha_{t}\right)\right)*T_j\\
&= \left(sh_j
-accum\_rec_{j}\right)*T_j
\end{aligned}
\end{equation}\]
\[\begin{equation}
\begin{aligned}
accum\_rec_{j} &= \sum_{k=j+1}sh_k\alpha_k\prod^{k-1}_{t=j+1}\left(1-\alpha_{t}\right)
\\
&= sh_{j+1}\alpha_{j+1} + sh_{j+2}\alpha_{j+2}(1-\alpha_{j+1}) + sh_{j+3}\alpha_{j+3}(1-\alpha_{j+1})(1-\alpha_{j+2}) + ...\\
&= \alpha_{j+1}sh_{j+1} + (1-\alpha_{j+1})*accum\_rec_{j+1}
\end{aligned}
\end{equation}\]
-
dL_dalpha
\(\begin{equation}
\frac{\partial L}{\partial \alpha_j} = \frac{\partial L}{\partial ch}\frac{\partial ch}{\partial \alpha_j}\\= \frac{\partial L}{\partial ch} *(sh_j - accum\_rec_j) * T_j + （background \ color \ part）
\end{equation}\)
2.1. If $\alpha_j = \left(1-\exp \left(-o_i\mathcal{G}_i \delta_i\right)\right)$
-
dL_dconic2D
\(\begin{equation}
\begin{aligned}
\frac{\partial L}{\partial \Sigma^{-1}} &= \frac{\partial L}{\partial ch}\frac{\partial ch}{\partial \alpha_j}\frac{\partial \alpha_j}{\partial G_j}\frac{\partial G_j}{\partial \Sigma^{-1}}
= \frac{\partial L}{\partial G_j}\frac{\partial G_j}{\partial \Sigma^{-1}}
\end{aligned}
\end{equation}\)
\[\begin{equation}
\begin{aligned}
\frac{\partial G_j}{\partial \Sigma^{-1}} &= \frac{\partial G_j}{\partial power_j}\frac{\partial power_j}{\partial \Sigma^{-1}}
\\
&= G_j\frac{\partial power_j}{\partial \Sigma^{-1}} = G_j(-0.5*dd^T)
\end{aligned}
\end{equation}\]
-
dL_do
\(\frac{\partial L}{\partial o_j} = \frac{\partial L}{\partial \alpha_j}\frac{\partial \alpha_j}{\partial o_j} = \frac{\partial L}{\partial \alpha_j} G\)
3.2. Volume Alpha: $\alpha_j = 1- \exp(-o_j * G_j * \delta_j)$
- dL_ddelta
\(\frac{\partial L}{\partial \delta_j} = \frac{\partial L}{\partial \alpha_j}\frac{\partial \alpha_j}{\partial \delta_j} = \frac{\partial L}{\partial \alpha_j} \exp(-o_j * G_j * \delta_j) * o_j * G_j\)
按照NeRF来说，这个梯度不需要传导，因为采样间隔应该是均匀的。但是也计算一下试试看。
\(\delta_j = d_{j+1} - d_j\)
In the backward pass, we assume the Gaussians are sorted in right order, but in the forward pass, we calculate $\delta_j = |d_{j+1} - d_j|$.
\(\frac{\partial \delta_j}{\partial d_j} = -1,
\frac{\partial \delta_j}{\partial d_{j+1}} = 1\)
- dL_dG
\(\frac{\partial L}{\partial G_j } = \frac{\partial L}{\partial \alpha_j}\frac{\partial \alpha_j}{\partial G_j } = \frac{\partial L}{\partial \alpha_j} \exp(-o_j * G_j * \delta_j) * o_j * \delta_j\)
- dL_do
\(\frac{\partial L}{\partial o_j} = \frac{\partial L}{\partial \alpha_j}\frac{\partial \alpha_j}{\partial o_j} = \frac{\partial L}{\partial \alpha_j} \exp(-o_j * G_j * \delta_j) *G_j * \delta_j\)
Ray-splat points
\[\mathbf{x}=(x z, y z, z, 1)^{\mathrm{T}}=\mathbf{W} P(u, v)=\mathbf{W H}(u, v, 1,1)^{\mathrm{T}}\]
\[\hat{\mathbf{x}}=(x, y, z)^{\mathrm{T}}=(\frac{1}{z}, \frac{1}{z}, 1)\mathbf{W} P(u, v)\]
Depth
Mean depth
\[D=\sum_i \omega_i z_i\]
where
\[\omega_i=T_i \alpha_i \hat{\mathcal{G}}_i(\mathbf{u}(\mathbf{x}))\]
is the weight contribution of the
$i$
-th Gaussian and
\[T_i=\prod_{j=1}^{i-1}\left(1-\alpha_j \hat{\mathcal{G}}_j(\mathbf{u}(\mathbf{x}))\right)\]
measures its visibility.
Calculate
\[z_i\]
\[\frac{\partial z_{\text {mean }}}{\partial z_i}=\omega_i\]
\[\frac{\partial L}{\partial z_i}=\frac{\partial L}{D}\frac{D}{\partial z_i}=\frac{\partial L}{D}\omega_i\]
Calculate
\[\omega_i\]
\[\frac{\partial z_{\text {mean }}}{\partial \omega_i}= z_i\]
\[\frac{\partial L}{\partial \omega_i}=\frac{\partial L}{D}\frac{D}{\partial \omega_i}=\frac{\partial L}{D}z_i\]
Calculate
\[\alpha_i\]
\[\frac{\partial T_i}{\partial \alpha_i} = T_{i-1} \left(-\hat{\mathcal{G}}_{i-1}(\mathbf{u}(\mathbf{x}))\right)\]
\[\frac{\partial \omega_i}{\partial \alpha_i}= T_i \hat{\mathcal{G}}_i(\mathbf{u}(\mathbf{x})) + \alpha_i \hat{\mathcal{G}}_i(\mathbf{u}(\mathbf{x})) \frac{\partial T_i}{\partial \alpha_i}\\
= T_i \hat{\mathcal{G}}_i(\mathbf{u}(\mathbf{x})) - \alpha_i T_{i-1} \hat{\mathcal{G}}_{i-1}(\mathbf{u}(\mathbf{x})) \hat{\mathcal{G}}_i(\mathbf{u}(\mathbf{x}))\]
Distortion loss
\[\begin{aligned}
&\begin{aligned}
\mathcal{Dist} & =\sum_{i=0}^{N-1} \sum_{j=0}^{i-1} \omega_i \omega_j\left(m_i-m_j\right)^2 \\
& =\sum_{i=0}^{N-1} \omega_i\left(m_i^2 \sum_{j=0}^{i-1} \omega_j+\sum_{j=0}^{i-1} \omega_j m_j^2-2 m_i \sum_{j=0}^{i-1} \omega_j m_j\right) \\
& =\sum_{i=0}^{N-1} \omega_i\left(m_i^2 A_{i-1}+D_{i-1}^2-2 m_i D_{i-1}\right)
\end{aligned}\\
&\text { where } A_i=\sum_{j=0}^{i} \omega_j, D_i=\sum_{j=0}^{i} \omega_j m_j \text { and } D_i^2=\sum_{j=0}^{i} \omega_j m_j^2 \text {. }
\end{aligned}\]
\[A_i=\sum_{j=0}^{i} \omega_j =\sum_{j=0}^{i} \alpha_j \hat{\mathcal{G}}_j(\mathbf{u}(\mathbf{x})) T_j = 1.0 - T_i = 1.0 - \prod_{j=1}^{i}\left(1-\alpha_j \hat{\mathcal{G}}_j(\mathbf{u}(\mathbf{x}))\right)\]
\[\frac{\partial \mathcal{Dist}}{\partial m_i}=\omega_i * 2 * m_i * A_{i-1} - 2 D_{i-1} = 2 (\omega_i * m_i * A_{i-1} - D_{i-1})\]
\[\frac{\partial \mathcal{Dist}}{\partial \omega_i}=m_i^2 A_{i-1}+D_{i-1}^2-2 m_i D_{i-1}\]
\[\frac{\partial \mathcal{Dist}}{\partial \alpha_i}=\frac{\partial L}{\partial \omega_i} \frac{\partial \omega_i}{\partial \alpha_i}\]]]>