在 [ ]

# Copyright (c) Meta Platforms, Inc. and affiliates. All rights reserved.

渲染彩色点云¶

本教程展示了如何

设置渲染器
渲染点云
改变渲染设置，例如合成和相机位置

导入模块¶

确保已安装torch和torchvision。如果未安装pytorch3d，请使用以下单元格安装它

在 [ ]

import os
import sys
import torch
need_pytorch3d=False
try:
    import pytorch3d
except ModuleNotFoundError:
    need_pytorch3d=True
if need_pytorch3d:
    if torch.__version__.startswith("2.2.") and sys.platform.startswith("linux"):
        # We try to install PyTorch3D via a released wheel.
        pyt_version_str=torch.__version__.split("+")[0].replace(".", "")
        version_str="".join([
            f"py3{sys.version_info.minor}_cu",
            torch.version.cuda.replace(".",""),
            f"_pyt{pyt_version_str}"
        ])
        !pip install fvcore iopath
        !pip install --no-index --no-cache-dir pytorch3d -f https://dl.fbaipublicfiles.com/pytorch3d/packaging/wheels/{version_str}/download.html
    else:
        # We try to install PyTorch3D from source.
        !pip install 'git+https://github.com/facebookresearch/pytorch3d.git@stable'

在 [ ]

import os
import torch
import torch.nn.functional as F
import matplotlib.pyplot as plt

# Util function for loading point clouds|
import numpy as np

# Data structures and functions for rendering
from pytorch3d.structures import Pointclouds
from pytorch3d.vis.plotly_vis import AxisArgs, plot_batch_individually, plot_scene
from pytorch3d.renderer import (
    look_at_view_transform,
    FoVOrthographicCameras, 
    PointsRasterizationSettings,
    PointsRenderer,
    PulsarPointsRenderer,
    PointsRasterizer,
    AlphaCompositor,
    NormWeightedCompositor
)

加载点云和对应的颜色¶

加载并创建一个点云对象。

点云是PyTorch3D中提供的一种独特的数据结构，用于处理不同大小的点云批次。

如果使用Google Colab运行此笔记本，请运行以下单元格获取点云数据并将其保存在路径data/PittsburghBridge中：如果在本地运行，数据已存在于正确的路径中。

在 [ ]

!mkdir -p data/PittsburghBridge
!wget -P data/PittsburghBridge https://dl.fbaipublicfiles.com/pytorch3d/data/PittsburghBridge/pointcloud.npz

在 [ ]

# Setup
if torch.cuda.is_available():
    device = torch.device("cuda:0")
    torch.cuda.set_device(device)
else:
    device = torch.device("cpu")

# Set paths
DATA_DIR = "./data"
obj_filename = os.path.join(DATA_DIR, "PittsburghBridge/pointcloud.npz")

# Load point cloud
pointcloud = np.load(obj_filename)
verts = torch.Tensor(pointcloud['verts']).to(device)
        
rgb = torch.Tensor(pointcloud['rgb']).to(device)

point_cloud = Pointclouds(points=[verts], features=[rgb])

创建渲染器¶

PyTorch3D中的渲染器由一个光栅化器和一个着色器组成，每个着色器都包含许多子组件，例如相机（正交/透视）。这里我们初始化其中一些组件，并对其余组件使用默认值。

在本例中，我们将首先创建一个使用正交相机并应用alpha合成的渲染器。然后我们将学习如何使用模块化API改变不同的组件。

[1] SynSin：从单张图像到端到端视图合成。 Olivia Wiles、Georgia Gkioxari、Richard Szeliski、Justin Johnson。CVPR 2020。

在 [ ]

# Initialize a camera.
R, T = look_at_view_transform(20, 10, 0)
cameras = FoVOrthographicCameras(device=device, R=R, T=T, znear=0.01)

# Define the settings for rasterization and shading. Here we set the output image to be of size
# 512x512. As we are rendering images for visualization purposes only we will set faces_per_pixel=1
# and blur_radius=0.0. Refer to raster_points.py for explanations of these parameters. 
raster_settings = PointsRasterizationSettings(
    image_size=512, 
    radius = 0.003,
    points_per_pixel = 10
)


# Create a points renderer by compositing points using an alpha compositor (nearer points
# are weighted more heavily). See [1] for an explanation.
rasterizer = PointsRasterizer(cameras=cameras, raster_settings=raster_settings)
renderer = PointsRenderer(
    rasterizer=rasterizer,
    compositor=AlphaCompositor()
)

在 [ ]

images = renderer(point_cloud)
plt.figure(figsize=(10, 10))
plt.imshow(images[0, ..., :3].cpu().numpy())
plt.axis("off");

我们现在将修改渲染器以使用具有设置的背景色的alpha合成。

在 [ ]

renderer = PointsRenderer(
    rasterizer=rasterizer,
    # Pass in background_color to the alpha compositor, setting the background color 
    # to the 3 item tuple, representing rgb on a scale of 0 -> 1, in this case blue
    compositor=AlphaCompositor(background_color=(0, 0, 1))
)
images = renderer(point_cloud)

plt.figure(figsize=(10, 10))
plt.imshow(images[0, ..., :3].cpu().numpy())
plt.axis("off");

在本例中，我们将首先创建一个使用正交相机并应用加权合成的渲染器。

在 [ ]

# Initialize a camera.
R, T = look_at_view_transform(20, 10, 0)
cameras = FoVOrthographicCameras(device=device, R=R, T=T, znear=0.01)

# Define the settings for rasterization and shading. Here we set the output image to be of size
# 512x512. As we are rendering images for visualization purposes only we will set faces_per_pixel=1
# and blur_radius=0.0. Refer to rasterize_points.py for explanations of these parameters. 
raster_settings = PointsRasterizationSettings(
    image_size=512, 
    radius = 0.003,
    points_per_pixel = 10
)


# Create a points renderer by compositing points using an weighted compositor (3D points are
# weighted according to their distance to a pixel and accumulated using a weighted sum)
renderer = PointsRenderer(
    rasterizer=PointsRasterizer(cameras=cameras, raster_settings=raster_settings),
    compositor=NormWeightedCompositor()
)

在 [ ]

images = renderer(point_cloud)
plt.figure(figsize=(10, 10))
plt.imshow(images[0, ..., :3].cpu().numpy())
plt.axis("off");

我们现在将修改渲染器以使用具有设置的背景色的加权合成。

在 [ ]

renderer = PointsRenderer(
    rasterizer=PointsRasterizer(cameras=cameras, raster_settings=raster_settings),
    # Pass in background_color to the norm weighted compositor, setting the background color 
    # to the 3 item tuple, representing rgb on a scale of 0 -> 1, in this case red
    compositor=NormWeightedCompositor(background_color=(1,0,0))
)
images = renderer(point_cloud)
plt.figure(figsize=(10, 10))
plt.imshow(images[0, ..., :3].cpu().numpy())
plt.axis("off");

使用pulsar后端¶

切换到pulsar后端很容易！pulsar后端内置了合成器，因此在创建时不需要compositor参数（如果您提供它，会显示警告）。它在渲染设备上预分配内存，因此在构建时需要n_channels。

渲染器正向函数的所有参数都是批次级的，除了背景颜色（在本例中为gamma）之外，您必须提供与批次中示例数量一样多的值。背景颜色是可选的，默认设置为全零。您可以在论文使用基于球体的表示进行神经渲染的快速可微光线投射中找到gamma如何影响渲染函数的详细解释。

您还可以对pulsar后端使用native后端，它已经提供了对点不透明度的访问。native后端可以从pytorch3d.renderer.points.pulsar导入；您可以在docs/examples文件夹中找到此方面的示例。

在 [ ]

renderer = PulsarPointsRenderer(
    rasterizer=PointsRasterizer(cameras=cameras, raster_settings=raster_settings),
    n_channels=4
).to(device)

images = renderer(point_cloud, gamma=(1e-4,),
                  bg_col=torch.tensor([0.0, 1.0, 0.0, 1.0], dtype=torch.float32, device=device))
plt.figure(figsize=(10, 10))
plt.imshow(images[0, ..., :3].cpu().numpy())
plt.axis("off");

在Plotly图形中查看点云¶

这里我们使用PyTorch3D函数plot_scene在Plotly图形中渲染点云。plot_scene返回一个plotly图形，其中轨迹和子图由输入定义。

在 [ ]

plot_scene({
    "Pointcloud": {
        "person": point_cloud
    }
})

我们现在将渲染一批点云。第一个点云与上面相同，第二个点云是全黑的，并且在所有维度上偏移了2，这样我们就可以在同一张图上看到它们。

在 [ ]

point_cloud_batch = Pointclouds(points=[verts, verts + 2], features=[rgb, torch.zeros_like(rgb)])
# render both in the same plot in different traces
fig = plot_scene({
    "Pointcloud": {
        "person": point_cloud_batch[0],
        "person2": point_cloud_batch[1]
    }
})
fig.show()

在 [ ]

# render both in the same plot in one trace
fig = plot_scene({
    "Pointcloud": {
        "2 people": point_cloud_batch
    }
})
fig.show()

对于批次，我们还可以使用plot_batch_individually来避免自己构建场景字典。

在 [ ]

# render both in 1 row in different subplots
fig2 = plot_batch_individually(point_cloud_batch, ncols=2)
fig2.show()

在 [ ]

# modify the plotly figure height and width
fig2.update_layout(height=500, width=500)
fig2.show()

我们还可以修改任一函数的轴参数和轴背景，并在plot_batch_individually中为我们的绘图添加标题。

在 [ ]

fig3 = plot_batch_individually(
    point_cloud_batch, 
    xaxis={"backgroundcolor":"rgb(200, 200, 230)"},
    yaxis={"backgroundcolor":"rgb(230, 200, 200)"},
    zaxis={"backgroundcolor":"rgb(200, 230, 200)"}, 
    subplot_titles=["Pointcloud1", "Pointcloud2"], # this should have a title for each subplot, titles can be ""
    axis_args=AxisArgs(showgrid=True))
fig3.show()