LION: Latent Point Diffusion Models for 3D Shape Generation

63bd0ac90074b.jpg

Background

  • Denoising Diffusion Models

    63bd0beda0c2a.png

  • Forward Process

    q(x1:T∣x0):=∏t=1Tq(xt∣xt−1),q(xt∣xt−1):=N(xt;1−βtxt−1,βtI)q\left(\mathbf{x}_{1: T} \mid \mathbf{x}_0\right):=\prod_{t=1}^T q\left(\mathbf{x}_t \mid \mathbf{x}_{t-1}\right), \quad q\left(\mathbf{x}_t \mid \mathbf{x}_{t-1}\right):=\mathcal{N}\left(\mathbf{x}_t ; \sqrt{1-\beta_t} \mathbf{x}_{t-1}, \beta_t \boldsymbol{I}\right)

    • TT : number of steps
    • q(xt∣xt−1)q(\mathbf{x}_t|\mathbf{x}_{t-1}) : Gaussian transition kernel, gradually adds noise to the input with a variance schedule β1,⋯ ,βT\beta_1,\cdots,\beta_T
    • The βt\beta_t are chosen such that the chain approximately converges to a standard Gaussian distribution after TT steps, q(xT)≈N(xT;0,I)q(\mathbf{x}_T)\approx \mathcal{N}(\mathbf{x}_T;\mathbf{0},\mathbf{I})
    • Property: sampling xt\mathbf{x}_t at an arbitrary timestep tt in closed form(αt:=1−βt\alpha_t:=1-\beta_t, αˉt:=∏s=1tαs\bar{\alpha}_t:=\prod_{s=1}^t \alpha_s): q(xt∣x0)=N(xt;αˉtx0,(1−αˉt)I)q\left(\mathbf{x}_t \mid \mathbf{x}_0\right)=\mathcal{N}\left(\mathbf{x}_t ; \sqrt{\bar{\alpha}_t} \mathbf{x}_0,\left(1-\bar{\alpha}_t\right) \mathbf{I}\right)
  • DDMs

    DDMs learn a parametrized reverse process(model parameter θ\theta) that inverts the forward diffusion: pθ(x0:T):=p(xT)∏t−1Tpθ(xt−1∣xt),pθ(xt−1∣xt):=N(xt−1;μθ(xt,t),ρt2I)p_{\boldsymbol{\theta}}\left(\mathbf{x}_{0: T}\right):=p\left(\mathbf{x}_T\right) \prod_{t-1}^T p_{\boldsymbol{\theta}}\left(\mathbf{x}_{t-1} \mid \mathbf{x}_t\right), \quad p_{\boldsymbol{\theta}}\left(\mathbf{x}_{t-1} \mid \mathbf{x}_t\right):=\mathcal{N}\left(\mathbf{x}_{t-1} ; \mu_{\boldsymbol{\theta}}\left(\mathbf{x}_t, t\right), \rho_t^2 \boldsymbol{I}\right)

    • NLL(Negative log likelihood) E[−log⁡pθ(x0)]≤Eq[−log⁡pθ(x0:T)q(x1:T∣x0)]=Eq[−log⁡p(xT)−∑t≥1log⁡pθ(xt−1∣xt)q(xt∣xt−1)]=:L\mathbb{E}\left[-\log p_\theta\left(\mathbf{x}_0\right)\right] \leq \mathbb{E}_q\left[-\log \frac{p_\theta\left(\mathbf{x}_{0: T}\right)}{q\left(\mathbf{x}_{1: T} \mid \mathbf{x}_0\right)}\right]=\mathbb{E}_q\left[-\log p\left(\mathbf{x}_T\right)-\sum_{t \geq 1} \log \frac{p_\theta\left(\mathbf{x}_{t-1} \mid \mathbf{x}_t\right)}{q\left(\mathbf{x}_t \mid \mathbf{x}_{t-1}\right)}\right]=: L L=Eq[DKL(q(xT∣x0)∥p(xT))⏟LT+∑t>1DKL(q(xt−1∣xt,x0)∥pθ(xt−1∣xt))⏟Lt−1−log⁡pθ(x0∣x1)⏟L0]L=\mathbb{E}_q[\underbrace{D_{\mathrm{KL}}\left(q\left(\mathbf{x}_T \mid \mathbf{x}_0\right) \| p\left(\mathbf{x}_T\right)\right)}_{L_T}+\sum_{t>1} \underbrace{D_{\mathrm{KL}}\left(q\left(\mathbf{x}_{t-1} \mid \mathbf{x}_t, \mathbf{x}_0\right) \| p_\theta\left(\mathbf{x}_{t-1} \mid \mathbf{x}_t\right)\right)}_{L_{t-1}} \underbrace{-\log p_\theta\left(\mathbf{x}_0 \mid \mathbf{x}_1\right)}_{L_0}]
    • Details

      L=Eq[−log⁡pθ(x0:T)q(x1:T∣x0)]=Eq[−log⁡p(xT)−∑t≥1log⁡pθ(xt−1∣xt)q(xt∣xt−1)]=Eq[−log⁡p(xT)−∑t>1log⁡pθ(xt−1∣xt)q(xt∣xt−1)−log⁡pθ(x0∣x1)q(x1∣x0)]=Eq[−log⁡p(xT)−∑t>1log⁡pθ(xt−1∣xt)q(xt−1∣xt,x0)⋅q(xt−1∣x0)q(xt∣x0)−log⁡pθ(x0∣x1)q(x1∣x0)]=Eq[−log⁡p(xT)q(xT∣x0)−∑t>1log⁡pθ(xt−1∣xt)q(xt−1∣xt,x0)−log⁡pθ(x0∣x1)]=Eq[DKL(q(xT∣x0)∥p(xT))+∑t>1DKL(q(xt−1∣xt,x0)∥pθ(xt−1∣xt))−log⁡pθ(x0∣x1)]\begin{aligned}L & =\mathbb{E}_q\left[-\log \frac{p_\theta\left(\mathbf{x}_{0: T}\right)}{q\left(\mathbf{x}_{1: T} \mid \mathbf{x}_0\right)}\right] \\& =\mathbb{E}_q\left[-\log p\left(\mathbf{x}_T\right)-\sum_{t \geq 1} \log \frac{p_\theta\left(\mathbf{x}_{t-1} \mid \mathbf{x}_t\right)}{q\left(\mathbf{x}_t \mid \mathbf{x}_{t-1}\right)}\right] \\& =\mathbb{E}_q\left[-\log p\left(\mathbf{x}_T\right)-\sum_{t>1} \log \frac{p_\theta\left(\mathbf{x}_{t-1} \mid \mathbf{x}_t\right)}{q\left(\mathbf{x}_t \mid \mathbf{x}_{t-1}\right)}-\log \frac{p_\theta\left(\mathbf{x}_0 \mid \mathbf{x}_1\right)}{q\left(\mathbf{x}_1 \mid \mathbf{x}_0\right)}\right] \\& =\mathbb{E}_q\left[-\log p\left(\mathbf{x}_T\right)-\sum_{t>1} \log \frac{p_\theta\left(\mathbf{x}_{t-1} \mid \mathbf{x}_t\right)}{q\left(\mathbf{x}_{t-1} \mid \mathbf{x}_t, \mathbf{x}_0\right)} \cdot \frac{q\left(\mathbf{x}_{t-1} \mid \mathbf{x}_0\right)}{q\left(\mathbf{x}_t \mid \mathbf{x}_0\right)}-\log \frac{p_\theta\left(\mathbf{x}_0 \mid \mathbf{x}_1\right)}{q\left(\mathbf{x}_1 \mid \mathbf{x}_0\right)}\right] \\& =\mathbb{E}_q\left[-\log \frac{p\left(\mathbf{x}_T\right)}{q\left(\mathbf{x}_T \mid \mathbf{x}_0\right)}-\sum_{t>1} \log \frac{p_\theta\left(\mathbf{x}_{t-1} \mid \mathbf{x}_t\right)}{q\left(\mathbf{x}_{t-1} \mid \mathbf{x}_t, \mathbf{x}_0\right)}-\log p_\theta\left(\mathbf{x}_0 \mid \mathbf{x}_1\right)\right]\\&=\mathbb{E}_q\left[D_{\mathrm{KL}}\left(q\left(\mathbf{x}_T \mid \mathbf{x}_0\right) \| p\left(\mathbf{x}_T\right)\right)+\sum_{t>1} D_{\mathrm{KL}}\left(q\left(\mathbf{x}_{t-1} \mid \mathbf{x}_t, \mathbf{x}_0\right) \| p_\theta\left(\mathbf{x}_{t-1} \mid \mathbf{x}_t\right)\right)-\log p_\theta\left(\mathbf{x}_0 \mid \mathbf{x}_1\right)\right]\end{aligned}

  • Directly tracing

    q(xt−1∣xt,x0)=N(xt−1;μ~t(xt,x0),β~tI)q\left(\mathbf{x}_{t-1} \mid \mathbf{x}_t, \mathbf{x}_0\right)=\mathcal{N}\left(\mathbf{x}_{t-1} ; \tilde{\boldsymbol{\mu}}_t\left(\mathbf{x}_t, \mathbf{x}_0\right), \tilde{\beta}_t \mathbf{I}\right) > where μ~t(xt,x0):=αˉt−1βt1−αˉtx0+αt(1−αˉt−1)1−αˉtxt\quad \tilde{\boldsymbol{\mu}}_t\left(\mathbf{x}_t, \mathbf{x}_0\right):=\frac{\sqrt{\bar{\alpha}_{t-1}} \beta_t}{1-\bar{\alpha}_t} \mathbf{x}_0+\frac{\sqrt{\alpha_t}\left(1-\bar{\alpha}_{t-1}\right)}{1-\bar{\alpha}_t} \mathbf{x}_t \quad and β~t:=1−αˉt−11−αˉtβt\quad \tilde{\beta}_t:=\frac{1-\bar{\alpha}_{t-1}}{1-\bar{\alpha}_t} \beta_t

Diffusion models and denoising antoencoders

  • Forward process and LTL_T
    • Ignore the fact that the forward process variances βt\beta_t are learnable by reparameterization and instead fix them to constrant.
    • LTL_T is a constant during training and can be ignored.
  • Reverse process and L1:T−1L_{1:T-1}
    • Analysis pθ(xt−1∣xt)=N(xt−1;μθ(xt,t),Σθ(xt,t))p_\theta\left(\mathbf{x}_{t-1} \mid \mathbf{x}_t\right)=\mathcal{N}\left(\mathbf{x}_{t-1} ; \boldsymbol{\mu}_\theta\left(\mathbf{x}_t, t\right), \mathbf{\Sigma}_\theta\left(\mathbf{x}_t, t\right)\right)
    • Set Σθ(xt,t)=σt2I\mathbf{\Sigma}_\theta\left(\mathbf{x}_t, t\right)=\sigma_t^2 \mathbf{I} to untrained time dependent constants Lt−1=Eq[12σt2∥μ~t(xt,x0)−μθ(xt,t)∥2]+CL_{t-1}=\mathbb{E}_q\left[\frac{1}{2 \sigma_t^2}\left\|\tilde{\boldsymbol{\mu}}_t\left(\mathbf{x}_t, \mathbf{x}_0\right)-\boldsymbol{\mu}_\theta\left(\mathbf{x}_t, t\right)\right\|^2\right]+C Lt−1−C=Eq[12σt2∥αˉt−1βt1−αˉtx0+αt(1−αˉt−1)1−αˉtxt−μθ(xt,t)∥2]=Ex0,ϵ[12σt2∥1αt(xt(x0,ϵ)−βt1−αˉtϵ)−μθ(xt(x0,ϵ),t)∥2]\begin{aligned}L_{t-1}-C & =\mathbb{E}_q\left[\frac{1}{2 \sigma_t^2}\left\|\frac{\sqrt{\bar{\alpha}_{t-1}} \beta_t}{1-\bar{\alpha}_t} \mathbf{x}_0+\frac{\sqrt{\alpha_t}\left(1-\bar{\alpha}_{t-1}\right)}{1-\bar{\alpha}_t} \mathbf{x}_t-\boldsymbol{\mu}_\theta\left(\mathbf{x}_t, t\right)\right\|^2\right] \\& =\mathbb{E}_{\mathbf{x}_0, \boldsymbol{\epsilon}}\left[\frac{1}{2 \sigma_t^2}\left\|\frac{1}{\sqrt{\alpha_t}}\left(\mathbf{x}_t\left(\mathbf{x}_0, \boldsymbol{\epsilon}\right)-\frac{\beta_t}{\sqrt{1-\bar{\alpha}_t}} \boldsymbol{\epsilon}\right)-\boldsymbol{\mu}_\theta\left(\mathbf{x}_t\left(\mathbf{x}_0, \boldsymbol{\epsilon}\right), t\right)\right\|^2\right]\end{aligned} μ\boldsymbol{\mu} must predict 1αt(xt−βt1−αˉtϵ)\frac{1}{\sqrt{\alpha_t}}\left(\mathbf{x}_t-\frac{\beta_t}{\sqrt{1-\bar{\alpha}_t}} \boldsymbol{\epsilon}\right) given xt\textbf{x}_t
    • Choose μθ(xt,t)=1αt(xt−βt1−αˉtϵθ(xt,t))\boldsymbol{\mu}_\theta\left(\mathbf{x}_t, t\right)=\frac{1}{\sqrt{\alpha_t}}\left(\mathbf{x}_t-\frac{\beta_t}{\sqrt{1-\bar{\alpha}_t}} \boldsymbol{\epsilon}_\theta\left(\mathbf{x}_t, t\right)\right) Ex0,ϵ[βt22σt2αt(1−αˉt)∥ϵ−ϵθ(αˉtx0+1−αˉtϵ,t)∥2]\mathbb{E}_{\mathbf{x}_0, \boldsymbol{\epsilon}}\left[\frac{\beta_t^2}{2 \sigma_t^2 \alpha_t\left(1-\bar{\alpha}_t\right)}\left\|\boldsymbol{\epsilon}-\boldsymbol{\epsilon}_\theta\left(\sqrt{\bar{\alpha}_t} \mathbf{x}_0+\sqrt{1-\bar{\alpha}_t} \boldsymbol{\epsilon}, t\right)\right\|^2\right]

Hierarchical Latent Point Diffusion Models(LION)

63bd0c4098c1f.png

Loss Function

min⁡θEt∼U{1,T},x0∼p(x0),ϵ∼N(0,I)[w(t)∥ϵ−ϵθ(αtx0+σtϵ,t)∥22],w(t)=βt22ρt2(1−βt)(1−αt2)\min _{\boldsymbol{\theta}} \mathbb{E}_{t \sim U\{1, T\}, \mathbf{x}_0 \sim p\left(\mathbf{x}_0\right), \boldsymbol{\epsilon} \sim \mathcal{N}(\mathbf{0}, I)}\left[w(t)\left\|\boldsymbol{\epsilon}-\boldsymbol{\epsilon}_{\boldsymbol{\theta}}\left(\alpha_t \mathbf{x}_0+\sigma_t \boldsymbol{\epsilon}, t\right)\right\|_2^2\right], \quad w(t)=\frac{\beta_t^2}{2 \rho_t^2\left(1-\beta_t\right)\left(1-\alpha_t^2\right)}

  • ωt\omega_t : often set to 1 (constant)
  • After training: xt−1=11−βt(xt−βt1−αt2ϵθ(xt,t))+ρtη\mathbf{x}_{t-1}=\frac{1}{\sqrt{1-\beta_t}}\left(\mathbf{x}_t-\frac{\beta_t}{\sqrt{1-\alpha_t^2}} \boldsymbol{\epsilon}_{\boldsymbol{\theta}}\left(\mathbf{x}_t, t\right)\right)+\rho_t \boldsymbol{\eta}

H-VAE Configuration

  • Point Clouds x∈R3×N\mathbf{x}\in \mathbb{R}^{3\times N}
  • Global shape latent z0∈RDz\mathbf{z}_0 \in \mathbb{R}^{D_{\mathbf{z}}}
  • Point cloud-structured latent h0∈R(3+Dh)×N\mathbf{h}_0 \in \mathbb{R}^{\left(3+D_{\mathbf{h}}\right) \times N}
  • h0\mathbf{h}_0 :  a latent point cloud consisting of NN points with xyz-coordinates in R3\mathbb{R}^3

Two Stage Training

  • First Stage : Training it as a regular VAE with standard Gaussian priors

    LELBO(ϕ,ξ)=Ep(x),qϕ(z0∣x),qϕ(h0∣x,z0)[log⁡pξ(x∣h0,z0)−λzDKL(qϕ(z0∣x)∣p(z0))−λhDKL(qϕ(h0∣x,z0)∣p(h0))]\begin{aligned}\mathcal{L}_{\mathrm{ELBO}}(\boldsymbol{\phi}, \boldsymbol{\xi}) & =\mathbb{E}_{p(\mathbf{x}), q_\phi\left(\mathbf{z}_0 \mid \mathbf{x}\right), q_\phi\left(\mathbf{h}_0 \mid \mathbf{x}, \mathbf{z}_0\right)}\left[\log p_{\boldsymbol{\xi}}\left(\mathbf{x} \mid \mathbf{h}_0, \mathbf{z}_0\right)\right. \\& \left.-\lambda_{\mathbf{z}} D_{\mathrm{KL}}\left(q_\phi\left(\mathbf{z}_0 \mid \mathbf{x}\right) \mid p\left(\mathbf{z}_0\right)\right)-\lambda_{\mathbf{h}} D_{\mathrm{KL}}\left(q_\phi\left(\mathbf{h}_0 \mid \mathbf{x}, \mathbf{z}_0\right) \mid p\left(\mathbf{h}_0\right)\right)\right]\end{aligned}

    • ϕ\boldsymbol\phi: encoder parameters
    • ξ\boldsymbol\xi : decoder parameters
  • Second Stage: train the latent DDMs on the latent encodings
    • Fix the VAE's encoder and decoder networks
    • Train two latent DDMs on the encoding z0\mathbf{z}_0 and h0\mathbf{h}_0 sampled from qϕ(z0∣x)q_\phi\left(\mathbf{z}_0 \mid \mathbf{x}\right) and qϕ(h0∣x,z0)q_\phi\left(\mathbf{h}_0 \mid \mathbf{x}, \mathbf{z}_0\right) LSMz(θ)=Et∼U{1,T},p(x),qϕ(z0∣x),ϵ∼N(0,I)∥ϵ−ϵθ(zt,t)∥22,LSMh(ψ)=Et∼U{1,T},p(x),qϕ(z0∣x),qϕ(h0∣x,z0),ϵ∼N(0,I)∥ϵ−ϵψ(ht,z0,t)∥22,\begin{aligned}\mathcal{L}_{\mathrm{SM}^{\mathbf{z}}}(\boldsymbol{\theta}) & =\mathbb{E}_{t \sim U\{1, T\}, p(\mathbf{x}), q_\phi\left(\mathbf{z}_0 \mid \mathbf{x}\right), \boldsymbol{\epsilon} \sim \mathcal{N}(\mathbf{0}, \boldsymbol{I})}\left\|\boldsymbol{\epsilon}-\boldsymbol{\epsilon}_{\boldsymbol{\theta}}\left(\mathbf{z}_t, t\right)\right\|_2^2, \\ \mathcal{L}_{\mathrm{SM}^{\mathrm{h}}}(\boldsymbol{\psi}) & =\mathbb{E}_{t \sim U\{1, T\}, p(\mathbf{x}), q_\phi\left(\mathbf{z}_0 \mid \mathbf{x}\right), q_\phi\left(\mathbf{h}_0 \mid \mathbf{x}, \mathbf{z}_0\right), \boldsymbol{\epsilon} \sim \mathcal{N}(\mathbf{0}, \boldsymbol{I})}\left\|\boldsymbol{\epsilon}-\boldsymbol{\epsilon}_{\boldsymbol{\psi}}\left(\mathbf{h}_t, \mathbf{z}_0, t\right)\right\|_2^2,\end{aligned}
Diffusion ModelPoint Cloud Generation
暂无评论

发送评论 编辑评论 


				

			

						

				

					

						

							

								
							

							
						

					

				

				

					

						

							

								
							

							
						

					

				

				

					

						

							

								
							

							
							
													

					

				

			

			
			

				

					
				

			

				

											

							
							
						

																				
					
											
						

	

		
|´・ω・)ノ
ヾ(≧∇≦*)ゝ
(☆ω☆)
（╯‵□′）╯︵┴─┴
￣﹃￣
(/ω＼)
∠( ᐛ 」∠)＿
(๑•̀ㅁ•́ฅ)
→_→
୧(๑•̀⌄•́๑)૭
٩(ˊᗜˋ*)و
(ノ°ο°)ノ
(´இ皿இ｀)
⌇●﹏●⌇
(ฅ´ω`ฅ)
(╯°A°)╯︵○○○
φ(￣∇￣o)
ヾ(´･ ･｀｡)ノ"
( ง ᵒ̌皿ᵒ̌)ง⁼³₌₃
(ó﹏ò｡)
Σ(っ °Д °;)っ
( ,,´･ω･)ﾉ"(´っω･｀｡)
╮(╯▽╰)╭
o(*////▽////*)q
＞﹏＜
( ๑´•ω•) "(ㆆᴗㆆ)
😂
😀
😅
😊
🙂
🙃
😌
😍
😘
😜
😝
😏
😒
🙄
😳
😡
😔
😫
😱
😭
💩
👻
🙌
🖕
👍
👫
👬
👭
🌚
🌝
🙈
💊
😶
🙏
🍦
🍉
😣
Source: github.com/k4yt3x/flowerhd
	

	

		
颜文字
Emoji
小恐龙
花!
	


									

			

			
			
		

	






上一篇
下一篇