# Home

wiki.hanzheteng.com

## Welcome!

This is a wiki website hosting my notes in research and development. They are&#x20;

1. often hard to find through simple web search, and
2. worth reviewing again if having the same issues.

In other words, I will forget if I don't write them down :)&#x20;

In the meantime, this wiki website aims to&#x20;

1. organize the knowledge base **systematically**, and
2. rephrase them as **concise** as possible.

However, since this wiki serves primarily for reviewing purpose (for what I have learned/summarized) rather than as tutorials, the content might be hard to understand if you are fresh new to it.&#x20;

Anyhow, I hope you can find it interesting and useful. For robotics tutorials and robot documentation, please see [UCR Robotics Wiki](https://ucr-robotics.readthedocs.io/en/latest/) website.&#x20;

## External Links

* [Hanzhe Teng's personal website](https://hanzheteng.com/)
* [UCR Robotics Wiki](https://ucr-robotics.readthedocs.io/en/latest/)
* [ARCS Lab at UCR](https://sites.google.com/view/arcs-lab)


# 3D Printer

TBD.


# Laser Cutter

### Introduction

3D Printer and Laser Cutter are the most commonly used equipment for fast prototyping and manufacturing. The former builds a model from scratch, while the latter chips away the excess materials. Therefore, the materials we put into laser cutter are often flat boards with certain thickness (e.g., acrylic sheet, woods, etc.).

The machine we have is [VLS 3.60/75](https://www.ulsinc.com/products/platforms/vls3-75) from [Universal Laser Systems](https://www.ulsinc.com/), with a processing area of 24 x 12 in (610 x 305 mm). It is also limited by the type of materials and maximum thickness. Please ensure the material can be found in the database (will discuss more).&#x20;

![The ventilator (left) and VLS 3.60/75 (right)](/files/-MatWAr7gn0KfxsPGL1u)

### Software

There are two types of software needed.&#x20;

* The software that can draw [vector graphics](https://en.wikipedia.org/wiki/Vector_graphics) (e.g., Adobe Illustrator, CorelDRAW) or can export/output vector graphics (e.g., AutoCAD, Solidworks). You will need to design the schematics in these software and print it using "VLS printer", just like printing a MS word document using a normal printer.&#x20;
* The Universal Control Panel (UCP) software to handle the schematics in printing queue (sent by vector graphics software) and send to the laser cutter after a few modifications. The software can be downloaded from [its official website](https://www.ulsinc.com/support/software-downloads). Choose `UCP Installer`, and then download. During the installation, select the corresponding model VLS 3.60/75.&#x20;

### Operations

#### Step One

Draw the vector graph using one of the vector graphics software, and print it using VLS 3.60/75 printer.&#x20;

* Ensure the **color** is set to pure red RGB (255, 0, 0) or CMYK (0, 100, 100, 0) for all lines to be cut. See user manual for advanced usage of color (e.g., blue for marks).&#x20;
* Ensure the **linewidth** is set to 0.001” (.0254 mm) or less. (Only vector graphics software can have lines as thin as this much.)
* If applicable, ensure the line is set to 100% opacity.

#### Step Two

Prepare the graph to be cut in UCP software.

* Open UCP software, select file in printing queue, and adjust the position of the graph (often close to upper left). You may also duplicate the graph using the button on the panel.&#x20;
* Click **Settings** to select the correct material of the board. Please make sure it is supported and can be found in Materials Database.&#x20;
* In the page of selected material, set **Material Thickness** to be the thickness of the board plus 0.05 mm or 0.025 in (e.g., 0.150 in for a 0.125 in board). The redundancy is to ensure the laser can always cut through the board with clean edge. This value may vary from machine to machine, depending on the flatness of processing area. (Only need to change thickness; leave others as default.)

![Choose correct material and set thickness](/files/-MatMDywP88KLVJngLyX)

#### Step Three

Load the material and perform calibration.

* Place the material on the cutting table of the laser cutter, close to upper left corner.&#x20;
* Connect the USB cable (from the laser cutter) to a Windows computer running UCP software, and then turn on the laser cutter **and the ventilator**. (Note: the machine can be turned on \[by the power button] only if the USB cable is connected and UCP software is running properly.)
* Calibrate cutting table and focus the laser system (because the thickness of the material can vary every time).&#x20;
  * Move the laser focal lens to somewhere above the material, and place the focus tool **on top of** the material while close to the laser focal lens, as shown in the image.&#x20;
  * Move the engraving table up or down in the Z axis as necessary until the wedge shaped notch at the top of the focus tool can be placed as shown with the edge of the carriage fitting in the corner of the notch. **Laser processing results are very sensitive to focus, so pay close attention to the fit of the focus tool.**&#x20;
  * In UCP software, switch to System tab and click Calibrate under Cutting Table. After calibration, go back to the Viewer tab and the Z axis will be set to 0.000.
  * Tips: If engraving table reaches its maximum limit (e.g., it stops at -0.100 in Z axis), click Calibrate for Cutting Table once and repeat the above process until the focus tool fits perfectly.

&#x20;            <img src="/files/-MatWfhxhh11huEKa77I" alt="" data-size="original">   <img src="/files/-MatPLUlhI7fVwv6yeJ4" alt="" data-size="original">

![Once focus tool fits, click the Calibrate button in this page.](/files/-MatMv3dq7PUyLvuC7J6)

#### Step Four

Double check settings and execute the task.

* Double check the graph to be cut is indeed within the area of the material. Click a couple of points (on the boundary of the graph) in UCP software, and observe the position of the laser focal lens above the material.&#x20;
* Press the Play button to start vector cutting. Better not to directly look at the cutting point as it may produce invisible lights harming your eyes. &#x20;

![In Focus mode, moving cursers to the boundary of the graph](/files/-MatMz46-P5ICoK-gNIy)

### References

* [VLS 3.60/75 User Manual](https://peoplevine.blob.core.windows.net/media/397/business/3598/VLS_Platform_User_Guide.pdf)
* Arizona State University 3D Print and Laser Cutter Lab [Instructions](https://3dprintlab.engineering.asu.edu/2018/04/universal-laser-cutter-vls-6-60/)
* [Laser Cutter Tutorial Babe 32x18](https://live-3d-print-lab.pantheonsite.io/wp-content/uploads/2018/04/Laser-Cutter-Tutorial-Babe-32x18.pdf)


# Motion Capture System

## Introduction

Motion Capture System (or MoCap in short) refers to the system that can capture the motion (movement) of people or objects.

It often consists of many optical cameras that can produce and receive infrared rays (the emitters are placed around the camera). By placing reflective markers on the object, the MoCap can uniquely identify and record the pose of objects at a very high frequency (\~100Hz to 400Hz).

## Marker Arrangement

#### Types of markers

* Passive markers (a ball-shape marker that can reflect infrared rays) are often used. However, there also exists another type, active markers, that can produce infrared rays on its own.
* Active markers require power supply from the robot, and hence are suitable for mid-to-large size of robots. They are often used together with a synchronization module, which can sync the emitting moment of active markers with the camera shutter of the MoCap.

#### How to put markers on an object?

* There should be at least 3 markers rigidly attached to an object, in order to compute a valid 3D pose. To improve redundancy, we often put 4 or 5 markers on the object, and make sure at each moment of the motion at least 3 markers are visible to cameras from different positions.&#x20;
* The markers shall not be placed in a singular position (e.g., co-planar, linear), and are supposed to be arranged in an **asymmetric pattern**. Only in this way can we compute a unique pose.
* Make sure the relative positions between markers are rigid, and it will not deform as the object moves; otherwise, this can result in tracking errors.

<figure><img src="/files/dLjYXLg02P9f08STLITY" alt="" width="375"><figcaption><p>Example of a symmetric, unfavorable arrangement (courtesy of Crazyswarm project)</p></figcaption></figure>

* In the above example, the three markers on the side of propellers form a symmetric pattern, which may cause ambiguity in the down side and the right side (in terms of the viewpoint of this image). In other words, cameras viewing from the down side may falsely recognize that they are viewing from the right side.

#### For advanced users

It is possible to use less than 3 markers to track the robot pose. If only 3D position (without orientation) is needed, it is also possible to track with a single marker.

* This can be achieved by accessing the raw data of the position of each marker, and applying the Iterative Closest Point algorithm on the user end.
* This is often used in robot swarms, where a unique marker arrangement for each robot is hard to find as the number of robots increases. Robots with similar marker arrangement can cause confusion to the MoCap system.
* It is still true that having 3 markers is the minimum requirement to form an object and compute a full 6-dof pose in the MoCap software. This implementation does not require the object pose computed by the MoCap software, and directly access the raw data of marker positions and apply a user-defined algorithm to track objects.

## Network Settings

There are at least two networks required in a MoCap system. One is the camera network (local, unidentified, no connection to Internet), and the other is the user or client network. An example network setup of a MoCap system is the following.

<figure><img src="/files/tnH0lNIl7jE728zm8JMT" alt=""><figcaption></figcaption></figure>

**Data flow in this network is the following**: Each camera captures markers/objects and sends this data to the MoCap PC via a switch in the camera network, and the PC runs MoCap software to process data, compute object poses, and send to the user end via Ethernet or WiFi in the user network.

On the user PC, we often run a ROS package to process the incoming data, convert them into ROS format, and publish ROS topics accordingly.

In our college, the MoCap PC is a machine managed by the IT department, and therefore this PC must have connection to the BCOE engineering network at all times. In this case, the MoCap PC is connecting to 3 networks at the same time. As most of the PCs are not shipped with 3 network cards by default, we need to purchase a network card separately and install it manually. (For example, we use the Intel I210-T1 network adapter.)

This is also to say that you need to have an BCOE engineering account in order to login to the MoCap machine. (Every BCOE student should have one; please work with the systems if you don't remember.)

## System Calibration

System calibration is needed in the following cases: 1) first use or the system is not in use for a long time, 2) whenever the relative poses between cameras are changed (e.g., someone accidentally kicked the tripod or twisted the camera).

After a successful calibration, the parameters can be saved for further use in the following days, until the next time you feel necessary, or you have a higher requirement on the accuracy.

#### Camera Extrinsic Calibration

This is to calibrate (compute) the relative pose between each and every camera.

To do this, a person needs to hold a wand (with predefined marker arrangement known to the manufacturer) and walk in the field. The person shall wave the wand and try to cover as much field of view of cameras as possible.

* In the software, the trajectories of the wand are recorded and visualized during the calibration process; you can see if past trajectories have covered most of the area in each camera's view.
* Note: Avoid wearing clothes or objects (watch, glasses, phone) with reflective materials. You may walk around in the MoCap volume and check if any part of your body has been falsely identified as a marker.
* Basic steps are: 1) in the MoCap software, click start calibration, 2) a person waving the wand in the volume, 3) in the MoCap software, click stop calibration or compute poses.

The theory behind this is triangulation-based localization and least-square estimation.

* In the calibration step, the relative distance between markers on the calibration tool is known, whereas the relative poses between cameras are variables to be estimated by least-square solutions.
* In normal use, the relative poses between cameras are known, and the poses of objects are variables to be estimated by minimizing the least-square re-projection errors from multiple cameras seeing the same object.

#### Set Ground Plane

Extrinsic parameters are constraints on the relative poses between cameras, but do not specify the origin of the coordinate system (without origin being set, these cameras are free objects in 3D space). Setting the ground plane is to tell the cameras where the origin of the MoCap volume is, adding one more constraint to all cameras.

This can be done by placing another calibration tool (with predefined marker arrangement known to the manufacturer, often in a triangular shape) on the center of the ground of the MoCap volume, and clicking a simple button in the MoCap software.

## MoCap Providers

There are many MoCap system providers in the world. The leading companies in the US are VICON and Optitrack, which are the two we often see in college.

They both provide specialized software in Windows systems on the MoCap PC, where we can create objects from selected markers, perform system calibration, record marker/object trajectories, and set data streaming settings. The appearance of the software may look different, but the basic functionalities they provide are the same.

On the user end, we use the corresponding ROS packages to convert the raw, low-level data packets in TCP/IP into ROS topics.

#### VICON

VICON does not have its official ROS package available. We often use the following two packages:

* <https://github.com/ethz-asl/vicon_bridge>  (pose published in `geometry_msgs::TransformStamped` format, without velocity estimation)
* <https://github.com/KumarRobotics/vicon>  (pose published in `nav_msgs::Odometry` format, with velocity estimation)

In both ROS packages, you only need to specify the name of the object you created in the VICON software on the MoCap PC.

#### OptiTrack

OptiTrack has its official ROS package available: <https://github.com/ros-drivers/mocap_optitrack>

You need to configure a yaml file in the ROS package similar to the following, where you need to specify the object ID (a unique integer number), which pose information to be published, and some settings regarding the software on the MoCap PC (e.g., network IP address).

```
rigid_bodies:
    '1':
        pose: Robot_1/pose
        pose2d: Robot_1/ground_pose
        child_frame_id: Robot_1/base_link
        parent_frame_id: world
optitrack_config:
        multicast_address: 224.0.0.1
```

## Debugging

#### There are some spikes in the tracking trajectory

* Regarding the spikes on the curve, it is likely that VICON lost the track for a moment during the "spikes". To debug this, you may enable recording (in the VICON software) during experiments, and check the playback (animation) to see if all markers are stable from the beginning to the end (visible, continuous, not shown as ghost markers, etc.)
* If markers are stable all the time (less likely), then check the ROS package that makes use of VICON data for pose estimation.
* If it is confirmed that markers are not stable enough, we can try the following to fix it.
  * Double check the marker configuration, and see if there is any possibility to lose visibility, rigidity, or asymmetric pattern during experiments.
  * Switch to grayscale mode in each camera, adjust the parameters (focal length, exposure, circle filtering algorithm, etc.) to make sure each marker can be viewed clearly, in proper size, with no overlapping with any other markers at all times.
  * Double check the valid capturing volume and if the quadrotor has been traveled to any spot out of this volume (minimum requirement: seen by 4 cameras). It can be tested by manually moving a few markers around and checking the playback. It is possible to adjust (finely tune) the field of view of each camera (i.e. to change the direction/angle each camera is facing towards) to enlarge this volume.


# Sensors


# RGB-D Cameras

| Camera     | Intel RealSense                                            | Stereolabs Zed          | MYNT EYE               | Astra                         |
| ---------- | ---------------------------------------------------------- | ----------------------- | ---------------------- | ----------------------------- |
| Lens       | One RGB + Two IR                                           | Two RGB                 | Two RGB (Quad IR)      | <p>One RGB</p><p>+ One IR</p> |
| Depth Tech | Stereo IR                                                  | Stereo                  | Stereo + IR (optional) | IR                            |
| Shutter    | <p>Global Shutter (Two IR)</p><p>Rolling Shutter (RGB)</p> | Rolling Shutter         | Global Shutter         | Rolling Shutter               |
| IMU        | Yes (D435i)                                                | Yes (Zed Mini or Zed 2) | Yes                    | No                            |
| Outdoor    | Yes                                                        | Yes                     | Yes                    | No                            |
| Comments   |                                                            | NVIDIA GPU required     | GPU optional           |                               |

* Intel RealSense D435i: <https://store.intelrealsense.com/buy-intel-realsense-depth-camera-d435i.html>
* Stereolabs: <https://www.stereolabs.com/>
* MYNT EYE: <https://www.mynteye.com/> (Note: this company has shutdown in early 2020.)
* Intel RealSense cameras sorted by depth tech (see [this blog](https://www.intelrealsense.com/beginners-guide-to-depth/) for more information)
  * Stereo IR: D400 series  (outdoor)
  * Structured or Coded Light: SR300 series  (indoor)
  * ToF or LiDAR: L515 series  (indoor)


# Velodyne LiDAR

### Resources

* [VLP-16 User Manual](https://velodynelidar.com/wp-content/uploads/2019/12/63-9243-Rev-E-VLP-16-User-Manual.pdf)
* ROS bridge: <http://wiki.ros.org/velodyne_pointcloud> (raw packets to PointCloud2)
* ROS message definition for [sensor\_msgs::PointCloud2](http://docs.ros.org/en/noetic/api/sensor_msgs/html/msg/PointCloud2.html)

### Facts to know (VLP-16)

* Multiple laser returns are possible from any single laser shot because of beam divergence. VLP-16 supports three laser return modes: Strongest, Last, and Dual.&#x20;
* Phase Locking feature can be enabled when multiple Velodyne sensors are used to prevent interference between each other.&#x20;
* The sensor's motor can be set to work between 300 RPM and 1200 RPM, in increments of 60 RPM. Setting to 600 RPM is equivalent to 10 Hz. The horizontal resolution will be changed accordingly (between 0.1 to 0.4 degrees) and the value for 10 Hz is 0.2 degree.
* All 16 lasers are fired and recharged every 55.296 μs. (16 lasers are fired one by one (16 × 2.304 μs) followed by an idle period of 18.43 μs.)
* The timestamp in the packet indicates the time of the first data point in the packet. The exact time for each data point can be computed by adding a time offset to the timestamp.
* In the raw data, the vertical angles for even Laser IDs (0, 2, 4, ..., 14) increments from -15° to -1° and the vertical angles for odd Laser IDs (1, 3, 5, ..., 15) increments from 1° to 15°. This has been rearranged by the ROS package, such that the value in the `ring` field increases as the vertical angle increases (i.e. ring ID 0-15 maps to angle -15° to 15°).
* Velodyne rotates in a clock-wise (CW) manner. A point cloud frame (16 scans in 360°) can begin/end at any azimuth (horizontal) angle. The border between the start/end of point cloud frames is also observed to rotate slowly in a clock-wise direction (due to a small range of overlap between the start/end of each frame).&#x20;

### VLP-16 Parameters

```
// VLP-16 constant parameters
const int N_SCAN = 16;
const int Horizon_SCAN = 1800;  // 1800 for 600 RPM (10Hz)
const float ang_res_x = 0.2;    // degree, 0.2 for 600 RPM (10Hz)
const float ang_res_y = 2.0;    // degree, fixed FoV
const float ang_bottom = 15.0+0.1;

/* Field information of sensor_msgs::PointCloud2 messages from Velodyne LiDAR
fields[]  name       offset   datatype    count
  0       x            0      7(FLOAT32)    1
  1       y            4      7(FLOAT32)    1
  2       z            8      7(FLOAT32)    1
  3       intensity    16     7(FLOAT32)    1
  4       ring         20     4(UINT16)     1
*/
```


# Zed Camera

## Zed Camera Installation

* CUDA device required (Jetson TX2, AGX Xavier, etc.)
* download [the latest SDK](https://www.stereolabs.com/developers/release/) and install
* git clone [zed-ros-wrapper](https://github.com/UCR-Robotics/zed-ros-wrapper) and catkin\_make

## Collect rosbag

```bash
roslaunch zed_wrapper zedm.launch
rosbag record \
    /tf \
    /tf_static \
    /zedm/zed_node/depth/camera_info \
    /zedm/zed_node/depth/depth_registered \
    /zedm/zed_node/imu/data \
    /zedm/zed_node/imu/data_raw \
    /zedm/zed_node/left/camera_info \
    /zedm/zed_node/left/image_rect_color \
    /zedm/zed_node/right/camera_info \
    /zedm/zed_node/right/image_rect_color \
    /zedm/zed_node/point_cloud/cloud_registered \
    /zedm/zed_node/pose \
    /zedm/zed_node/pose_with_covariance \
    /zedm/zed_node/odom \
    /zedm/zed_node/left_cam_imu_transform

roslaunch zed_wrapper zed2.launch
rosbag record \
    /tf \
    /tf_static \
    /zed2/zed_node/depth/camera_info \
    /zed2/zed_node/depth/depth_registered \
    /zed2/zed_node/imu/data \
    /zed2/zed_node/imu/data_raw \
    /zed2/zed_node/left/camera_info \
    /zed2/zed_node/left/image_rect_color \
    /zed2/zed_node/right/camera_info \
    /zed2/zed_node/right/image_rect_color \
    /zed2/zed_node/point_cloud/cloud_registered \
    /zed2/zed_node/pose \
    /zed2/zed_node/pose_with_covariance \
    /zed2/zed_node/odom \
    /zed2/zed_node/left_cam_imu_transform

```


# RealSense D435i

## Ubuntu Installation

* Install `librealsense2` according to [this official GitHub repository](https://github.com/IntelRealSense/librealsense/blob/master/doc/distribution_linux.md#installing-the-packages)&#x20;
  * make sure install `librealsense2-dev` as well because it will provide cmake support
  * do not use `ros-melodic-librealsense2` library
* Install `ddynamic_reconfigure` by `sudo apt install ros-melodic-ddynamic-reconfigure`
* git clone [`realsense-ros` package](https://github.com/UCR-Robotics/realsense-ros) to workspace and catkin\_make

#### Notes for Jetson devices

* For Jetson devices, there is no `librealsense2-dkms` package available; only need to install `librealsense2-utils` and `librealsense2-dev`. ([installation\_jetson](https://github.com/IntelRealSense/librealsense/blob/master/doc/installation_jetson.md))
* During building process, `cv_bridge` expects `/usr/include/opencv` but on Jetson systems it's under `/usr/include/opencv4/opencv2`. To resolve this, create a symbolic link to the directories `sudo ln -s /usr/include/opencv4/ /usr/include/opencv` .

## Collect rosbag

```bash
roslaunch realsense2_camera rs_camera.launch filters:=pointcloud
rosbag record \
    /tf \
    /tf_static \
    /camera/color/camera_info \
    /camera/color/image_raw \
    /camera/depth/camera_info \
    /camera/depth/image_rect_raw \
    /camera/extrinsics/depth_to_color \
    /camera/depth/color/points  # this topic enabled by filters:=pointcloud
# rosbag size: 4GB/min
```


# IMU

### **MPU-9250**

* cheap, readily available low-end IMU, at the cost about a few dollars.
* Accelerometer noise density: 300 µg/√Hz
* Gyroscope noise density: 0.01 º/sec/√Hz

### **Microstrain 3DM-GX5**

* Industrial grade IMU made by microstrain; also available from Clearpath at about $2000.
* 3DM-GX5-25 is adopted by LIO-SAM project. Note that its accelerometer and gyroscope have different coordinate setup in orientation. (But still, they can be regarded to have the same origin.) For more information, see <https://github.com/TixiaoShan/LIO-SAM#prepare-imu-data>.
* We use 3DM-GX5-AHRS, and in this version the coordinate frames of accelerometer and gyroscope are treated identical. It comes equipped with built-in complementary filter and EKF algorithms, running at 1000Hz and 500Hz respectively.&#x20;
  * Accelerometer noise density: 20 µg/√Hz (2 g)&#x20;
  * Gyroscope noise density: 0.005°/sec/√Hz (300°/sec)

### VN-100

* High-end calibrated IMU with an onboard processor, provided by vectornav at about $1000.
* Adopted by the CoSTAR team in the SubT challenge.
* <https://www.vectornav.com/products/detail/vn-100>
* <https://github.com/NeBula-Autonomy/nebula-odometry-dataset/blob/main/pages/overview.md>

### **IMU Calibration Tool**

* Kalibr requires the noise parameters of IMU as input, and this can be calibrated using the Allan variance method. Open source repo: <https://github.com/ori-drs/allan_variance_ros>

```
# This is the result calibrated for the Microstrain 3DM-GX5-AHRS IMU that we have
# Accelerometer
accelerometer_noise_density: 0.0002276846771973415 
accelerometer_random_walk: 6.706087584689249e-06 

# Gyroscope
gyroscope_noise_density: 0.00010426654551109309 
gyroscope_random_walk: 3.5636559575381104e-06 

rostopic: '/sensors/imu' #Make sure this is correct
update_rate: 200.0 #Make sure this is correct
```


# eGPU

## Installation

In the following, we will show an example of installing eGPU (NVIDIA GTX Titan X + Razer Core X enclosure) on a Intel NUC computer with Ubuntu 18.04 operating system.&#x20;

In summary, there are four steps.&#x20;

* Follow [the official user manual](https://dl.razerzone.com/master-guides/RazerSynapse3/COREX-EN.pdf) provided by Razer Core X for hardware installation (or see [this video](https://www.youtube.com/watch?v=cVdymlfq2-8)).&#x20;
* Before installing NVIDIA drivers, make sure you can see the NVIDIA graphic card in PCI device list by command `lspci`. If not, go to BIOS settings, make sure Thunderbolt Controller is enabled, and Thunderbolt Security is disabled or in Legacy Mode.
* Once the graphic card is detected, we can proceed to install NVIDIA drivers. Open System Settings --> Software & Updates --> Additional Drivers, select **the best compatible one** for NVIDIA drivers and click Apply Changes button. (You may try different options and Apply Changes, until one of it finally works.)
* Once the installation is finished, reboot the computer. You may also want to reconnect the monitor to one of the display ports on eGPU (instead of the HDMI port on NUC we used previously).

For more details, please continue to read the following debugging log.&#x20;

## Debugging Log

### Thunderbolt Connection

* At the beginning, I was not able to see the graphic card in PCI device list (by command `lspci`). This has been resolved by switching to Legacy Mode for Thunderbolt Security Level in Intel BIOS settings. (In some BIOS settings, there might be an option to disable Thunderbolt Security, which is equivalent to the Legacy Mode.)
* In details, we can run command `lspci` to check PCI devices, and it is expected to see both part one and part two.
* ```
  # part one: indicate if the thunderbolt controller is working
  02:00.0 PCI bridge: Intel Corporation JHL6340 Thunderbolt 3 Bridge (C step) [Alpine Ridge 2C 2016] (rev 02)
  03:00.0 PCI bridge: Intel Corporation JHL6340 Thunderbolt 3 Bridge (C step) [Alpine Ridge 2C 2016] (rev 02)
  03:01.0 PCI bridge: Intel Corporation JHL6340 Thunderbolt 3 Bridge (C step) [Alpine Ridge 2C 2016] (rev 02)
  03:02.0 PCI bridge: Intel Corporation JHL6340 Thunderbolt 3 Bridge (C step) [Alpine Ridge 2C 2016] (rev 02)
  04:00.0 System peripheral: Intel Corporation JHL6340 Thunderbolt 3 NHI (C step) [Alpine Ridge 2C 2016] (rev 02)
  6c:00.0 USB controller: Intel Corporation Device 15db (rev 02)

  # part two: indicate if the NVIDIA card is detected
  05:00.0 PCI bridge: Intel Corporation JHL6340 Thunderbolt 3 Bridge (C step) [Alpine Ridge 2C 2016] (rev 02)
  06:01.0 PCI bridge: Intel Corporation JHL6340 Thunderbolt 3 Bridge (C step) [Alpine Ridge 2C 2016] (rev 02)
  07:00.0 VGA compatible controller: NVIDIA Corporation GP102 [TITAN X] (rev a1)
  07:00.1 Audio device: NVIDIA Corporation GP102 HDMI Audio Controller (rev a1)
  ```
* If you cannot see part one, make sure the thunderbolt controller is enabled in BIOS or somewhere else. If you cannot see part two, you might need to go to the BIOS settings to disable Thunderbolt Security, or set it to Legacy Mode.&#x20;

### NVIDIA Driver

* There are two approaches to install NVIDIA drivers. The first one is recommended and is the safest, since it keeps the integrity of Ubuntu package management software. However, if it doesn't work for you, you can try the second approach. (In other words, approach one is to install from Ubuntu official repository, and approach two is to install from NVIDIA official repository.)

**Approach One**

* Open System Settings --> Software & Updates --> Additional Drivers, and select **the best compatible** NVIDIA driver (latest is not necessarily the best) and click Apply Changes button. Once the installation is finished, reboot the computer and check if it works.&#x20;
* In my case, probably due to the old graphic card (GTX Titan released in 2015) or the OS (Ubuntu 18 instead of 20), the latest NVIDIA driver didn't work for me. I have tried 460, 450, and 418. Neither works. The xorg option is an open source solution provided by Linux community (not NVIDIA), and it not recommended. In the end, only the 390 driver works for me.&#x20;

{% tabs %}
{% tab title="nvidia-driver-390" %}
![](/files/-MYv0fGtNyx-BEdtalT2)
{% endtab %}

{% tab title="nvidia-driver-460" %}
![](/files/-MYs9mT4XdQETHS64xb3)
{% endtab %}
{% endtabs %}

* The issues I had during this process are the following. With the latest 460 driver, I can see the NVIDIA graphic card connected by command `nvidia-smi` . However, it is always idle there and I was not able to ask Ubuntu to switch to this graphic card. Also, I had to connect eGPU after booting Ubuntu, otherwise it will pause at the booting step with an error `iwlwifi BIOS contains WGDS but no WRDS` or simply blinking cursor. (To proceed, I had to unplug eGPU from thunderbolt port and connect it again after booting.)
* I have also tried to create a new file in `/etc/X11/xorg.conf.d/nvidia.conf` to force the system to use NVIDIA card. However, this turned out to be a dangerous operation and it made me not able to enter the system again. I guess this overwrote the default display option of Intel integrated graphic card in the booting step. In the end, I had to boot from a USB device to delete this file.&#x20;

#### Approach Two (not recommended; use as the last resort)

* Go to [NVIDIA official website](https://www.nvidia.com/Download/index.aspx) to download the corresponding driver installer. (As mentioned before, better to install the best compatible version, as the latest is not necessarily the best.)
* As the requirement of the installer, we have to exit X-server and keep only the terminal up running without GUI. See [this post](https://askubuntu.com/a/149224) for more information. Then run the installer using `sudo` privilege.&#x20;
* The above two steps are collected from the Internet. I haven't tried either and therefore I am not sure what issues may happen.&#x20;

### Tools

* To check which graphic card is currently in use. One option is to open System Settings --> Details (or About This Computer), and check the Graphics item. The other option is to use command `glxinfo`.
* ```
  glxinfo | grep -e "OpenGL vendor" -e "OpenGL renderer"
  // then you can see the following
  OpenGL vendor string: NVIDIA Corporation
  OpenGL renderer string: TITAN X (Pascal)/PCIe/SSE2
  ```
* Once the NVIDIA driver is installed, we can use command `nvidia-smi` to check the status of the graphic card. &#x20;

{% tabs %}
{% tab title="NVIDIA Card In Use" %}
![](/files/-MYv0fGx-9IQ_IeuOKMi)
{% endtab %}

{% tab title="NVIDIA Card Not In Use" %}
![](/files/-MYs9mT0t8eudEb_Ujm9)
{% endtab %}
{% endtabs %}

* For the record, the alternative command-line command for driver updates in Ubuntu is `sudo ubuntu-drivers devices` . In this case, we can see that `nvidia-driver-460` is the recommended driver, though this latest/recommended driver is not compatible in our case as mentioned before. &#x20;

<div align="center"><img src="/files/-MYs9mT3wgEVDQsqIP26" alt=""></div>


# Nvidia AGX Xavier

## Introduction

[Nvidia Jetson AGX Xavier](https://developer.nvidia.com/embedded/jetson-agx-xavier-developer-kit) was released in 2018 and is the most powerful onboard computer so far. In early 2020, Nvidia has doubled its RAM (from 16G to 32G) at the same price. Some notable features are as follows.

* 8-core ARMv8 CPU
* 512 CUDA cores
* 32GB DDR RAM
* 32GB eMMC storage
* possible to add 1TB NVMe SSD
* acceptable weight (700g)
* price $699

See [initial setup video](https://www.youtube.com/watch?v=-nX8eD7FusQ) and [unboxing video](https://www.youtube.com/watch?v=l5vOD1FenN8) for more information regarding interfaces and features. See [user manual](https://developer.download.nvidia.com/assets/embedded/secure/jetson/xavier/docs/jetson_agx_xavier_developer_kit_user_guide.pdf?J1ax1mCcys5raHxk4NxpNmgOgTxCScXAzsHfwslDWfJQvoCu_9HjpzLXXQKO4Sh6cuwS7OBj-0wijeGok_5kpn5G0T1w5qET2AYWDPH2E-MTkbxKdzyfl4A7jnJcGj0YN2SQBTUWx4_qFeOgpCgQ9bzxS1uwXj_WTHWrWzFZPUkM9iPSge6dHIfQqCi_D9keEAord8XtkA) for more details.&#x20;

Purchase List:&#x20;

* [Nvidia AGX Xavier (32G)](https://www.amazon.com/NVIDIA-Jetson-Xavier-Developer-32GB/dp/B083ZL3X5B/)
* [1TB Samsung 970 EVO SSD](https://www.amazon.com/Samsung-970-EVO-1TB-MZ-V7E1T0BW/dp/B07BN217QG/)
* [8265AC WiFi and Bluetooth module with antenna](https://www.amazon.com/Makeronics-Wireless-Bluetooth-Assembly-Instruction/dp/B07X2NLL85)

## Installation

It is required to have another Ubuntu 16 or 18 system to help flash OS image into AGX Xavier. It is recommended to use natively installed Ubuntu, but I have successfully flashed AGX Xavier using a Ubuntu 18 VM in VMware.&#x20;

Steps to install the system:&#x20;

* In host Ubuntu system (16 or 18), download and install [Nvidia SDK Manager](https://developer.nvidia.com/nvidia-sdk-manager). You may need to register an account in order to have access to the download page.&#x20;
* Follow [initial setup video](https://www.youtube.com/watch?v=-nX8eD7FusQ) or [user manual](https://developer.download.nvidia.com/assets/embedded/secure/jetson/xavier/docs/jetson_agx_xavier_developer_kit_user_guide.pdf?J1ax1mCcys5raHxk4NxpNmgOgTxCScXAzsHfwslDWfJQvoCu_9HjpzLXXQKO4Sh6cuwS7OBj-0wijeGok_5kpn5G0T1w5qET2AYWDPH2E-MTkbxKdzyfl4A7jnJcGj0YN2SQBTUWx4_qFeOgpCgQ9bzxS1uwXj_WTHWrWzFZPUkM9iPSge6dHIfQqCi_D9keEAord8XtkA) to set up cable connections. (refer to hardware connections only; the installer in setup video is no longer available; we use sdkmanager instead.)
* Press force recovery button and power button on AGX Xavier to enter recovery mode. To verify, run `lsusb` command on host Ubuntu and see if a Nvidia device is available. (Only in recovery mode can Xavier be detected as a device via USB connection.)
* Follow the instructions on SDK Manager and finish installation. Note that SDK Manager tries to install everything at once. However, I recommend to decompose it into three steps.&#x20;
  * flash Jetson OS first (uncheck SDK components; network connection on Xavier is not required at this step)
  * once finished, unplug USB connection, restart Xavier in normal mode, connect mouse, keyboard and monitor and set up this new Ubuntu system (username, password, time zone, etc.)
  * connect Xavier and host Ubuntu to the same network, restart SDK Manager on host Ubuntu and install Jetson SDK Components only (uncheck Jetson OS)

![](/files/-MHd_luigXj1g7o8ziYp)

## Post Installation

As introduced in the [unboxing video](https://www.youtube.com/watch?v=l5vOD1FenN8), Nvidia AGX Xavier by default runs at 15W balance mode with 4 core CPU. To check status and switch model, run the following code.

```
sudo nvpmodel -q     # check status
sudo nvpmodel -m 0   # switch to max performance mode
```

Next, we can proceed to install SSD storage and move our entire OS from eMMC to the new SSD storage.&#x20;

To install NVMe SSD, see [this video](https://youtu.be/x0TBTYw7HKs) for hardware instructions. Software instructions may differ in the last a couple steps. I recommend leaving 32GB unallocated for swap.

See [this page](https://www.seeedstudio.com/blog/2020/06/22/boot-jetson-xavier-from-m-2-ssd/) and this [github repo](https://github.com/jetsonhacks/rootOnNVMe) for instructions regarding moving root filesystem from eMMC to SSD storage. The original discussion on Nvidia forum can be seen [here](https://forums.developer.nvidia.com/t/how-to-boot-from-nvme-ssd/65147). In summary, the commands are as follows.

```
// make sure SSD is formatted to ext4, and a partition (p1) is set
git clone https://github.com/jetsonhacks/rootOnNVMe.git
cd rootOnNVMe
./copy-rootfs-ssd.sh
./setup-service.sh
sudo reboot
```

Lastly, you may want to install the WiFi + Bluetooth module. Hardware installation is obvious I believe. Software side only the following command is required, though they should be already installed. See also [this page](https://forums.developer.nvidia.com/t/intel-dual-band-wireless-ac-8265-on-xavier-solved/65464).&#x20;

```
sudo apt install linux-firmware
```


# CPU Benchmark

Notes:&#x20;

* Results are obtained from 2-3 runs; standard deviations are not reflected here (but mostly <5%). This is only for a rough estimate of the performance of each machine.
* For Blowfish, Fibonacci, N-Queens, FFT and Raytracing **lower is better** (the results are in seconds). For the rest, **higher is better**.&#x20;

| Computer           | <p>Raspberry Pi 3</p><p>ARM A53 (1GB)</p> | <p>Raspberry Pi 4</p><p>ARM A72 (4GB)</p> | UpBoard |
| ------------------ | ----------------------------------------- | ----------------------------------------- | ------- |
| CPU Blowfish (s)   | 42.49                                     | 6.97                                      | 6.54    |
| CPU CryptoHash     | 24.86                                     | 196.25                                    | 104.68  |
| CPU Fibonacci (s)  | 13.53                                     | 2.07                                      | 6.64    |
| CPU N-Queens (s)   | 37.46                                     | 9.13                                      | 24.87   |
| CPU Zlib           | /                                         | 0.29                                      | /       |
| FPU FFT (s)        | 47.19                                     | 5.82                                      | 6.07    |
| FPU Raytracing (s) | 42.30                                     | 2.62                                      | 20.77   |
| #events per sec    | 63.41                                     | 1487.04                                   | 420.34  |

| Computer           | <p>Jetson Nano<br>ARM A57 (4GB)</p> | <p>Jetson Xavier NX</p><p>ARMv8 (8GB)</p> | <p>Jetson Xavier AGX</p><p>ARMv8 (32GB)</p> |
| ------------------ | ----------------------------------- | ----------------------------------------- | ------------------------------------------- |
| CPU Blowfish (s)   | 7.51                                | 0.93                                      |                                             |
| CPU CryptoHash     | 202.06                              | 206.70                                    |                                             |
| CPU Fibonacci (s)  | 2.53                                | 1.30                                      |                                             |
| CPU N-Queens (s)   | 9.94                                | 74.38                                     |                                             |
| CPU Zlib           | 0.30                                | 0.46                                      |                                             |
| FPU FFT (s)        | 4.06                                | 0.59                                      |                                             |
| FPU Raytracing (s) | 4.57                                | 26.69                                     |                                             |
| #events per sec    | 1160.25                             | 1100.30                                   |                                             |

| Computer           | <p>Intel NUC</p><p>i7-8559U</p> | <p>Dell Precision</p><p>i7-?</p> | <p>Surface Book3</p><p>i7-1065G7 (VM)</p> |
| ------------------ | ------------------------------- | -------------------------------- | ----------------------------------------- |
| CPU Blowfish (s)   | 1.00                            | 0.69                             | 1.69                                      |
| CPU CryptoHash     | 1008.76                         | 1523.11                          | 806.20                                    |
| CPU Fibonacci (s)  | 0.41                            | 0.42                             | 0.52                                      |
| CPU N-Queens (s)   | 4.79                            | 5.81                             | 7.71                                      |
| CPU Zlib           | 1.51                            | 2.03                             | 0.48                                      |
| FPU FFT (s)        | 0.63                            | 0.60                             | 1.20                                      |
| FPU Raytracing (s) | 1.24                            | 1.36                             | 1.65                                      |
| #events per sec    | 1592.46                         | 1518.85                          | 2870.30                                   |

&#x20;Details about experiments:

* Blowfish, CryptoHash, Fibonacci, N-Queens, Zlib are tested by `hardinfo`, while the number of events per second is tested by `sysbench`.
* Jetson series are tested under the best performance mode.&#x20;
* To install, run `sudo apt install hardinfo sysbench`.
* To test using `hardinfo`, just do it by GUI. To test using `sysbench`, run `sysbench --test=cpu run`.&#x20;


# Installation Checklist

## Common

* set up SSH key and GitHub authorization (if it's personal computer)
* set up local Git user, email and alias information
* disable daily software upgrade (avoid race condition on apt lock)
* install ROS and create a new ROS workspace `catkin_ws`
* install VS Code IDE
  * x86-64 architecture: install from Ubuntu software center or download from [official website](https://code.visualstudio.com/)
  * arm architecture: download customized deb (latest release) from [this GitHub repository](https://github.com/toolboc/vscode)
  * after installation, add ROS, CMake extensions
* install terminator and set up config (see [this repo](https://github.com/hanzheteng/code_browsing)), set scroll-back lines to 10000 as in Gnome
* install commonly used software as follows

```
sudo apt install vim nmap net-tools exfat-fuse exfat-utils 
sudo apt install openssh-server xrdp hardinfo xdotool tmux tmuxp
sudo apt install git chromium-browser simplescreenrecorder
sudo apt install python3-catkin-tools
```

* set up shell alias (a few examples as follows)

```
alias max="xdotool key ctrl+super+Up"
alias four="max && xdotool key ctrl+shift+e && xdotool key ctrl+shift+o && xdotool key alt+Left && xdotool key ctrl+shift+o && xdotool key alt+Up"
alias jac="ssh -X administrator@jackal"
alias jacw="ssh -X administrator@jackalw"
```

* Install Slack, Zoom, Teamviewer, CloudCompare, SimpleScreenRecorder, etc. (GNOME has a default screenshot software.)
* Set up CloudCompare and Open3D Viewer as the default program to open `.pcd` files.
* Install languages: settings -> manage installed languages -> install new language -> reboot -> add input sources (intelligent pinyin).&#x20;

## Ubuntu 18

* add "new file" option to right click menu
  * by command `touch ~/Templates/"Untitled Document"`
* fix Gazebo ignition bug&#x20;
  * open file `~/.ignition/fuel/config.yaml` and replace the url inside from `https://api.ignitionfuel.org` to `https://api.ignitionrobotics.org`.
* Install Unity desktop ([link](https://askubuntu.com/questions/1124712/can-i-use-ubuntu-18-04-with-unity))
  * `sudo apt install ubuntu-unity-desktop`&#x20;
  * choose `lightdm` display manager
  * use `sudo dpkg-reconfigure lightdm` to reset display manager if needed
  * `sudo shutdown -r now` to restart the computer

## Ubuntu 20

* enable scale factor in display

```shell
sudo apt install python-is-python3  # for python in /usr/bin/env
```

## VS Code

* settings
  * lock to launch bar
  * window zoom level
  * editor font size
  * editor ruler set to 120
  * add folders to workspace
  * enable Wrap Tabs
* extensions
  * Github Co-Pilot
  * ROS (include C/C++ and Python)
  * CMake (for highlight)
  * CMake Tools (for C++ IntelliSense)
    * set "configurationProvider": "ms-vscode.cmake-tools"
    * run command CMake: Configure
    * add folder to .gitignore
  * Markdown Preview GitHub Styling `Ctrl + Shift + V`
  * reStructuredText
  * Remote SSH ([double SSH setup](https://blog.csdn.net/daqianc/article/details/122855698))
  * Doxygen Documentation Generator
  * Better C++ Syntax
* shortcuts
  * open terminal by default set to `` Ctrl + ` ``
  * maximize/minimize terminal window
    * set "View: Toggle Maximized Panel" to `Ctrl + Shift + M`
  * split terminal window
    * set "Terminal: Split Terminal" to `Ctrl + Shift + E`
* comments
  * C++ IntelliSense jump back to function `Ctrl + Alt + -`
  * C++ IntelliSense is not working on arm architecture

## Dual Boot on XPS 13

Tested on Dell XPS Plus 13 (9320) with Windows 11 OS pre-installed, and BIOS version 2.3.0.

Mostly following this article to install: <https://infra.engineer/miscellaneous/3-how-to-dual-boot-windows-10-and-ubuntu-on-dell-xps-13>

Although there are some differences in BIOS interface and Win11/Win10, basic steps are the same.&#x20;

Operations in Windows 11:

* Shrink Windows volume to create unallocated space for Ubuntu, using Disk Management.
* Turn off BitLocker or Drive Encryption.
* Turn off Fastboot (go to control panel, power settings, "choose what the power buttons do", and then uncheck "Turn on fast startup (Recommended)")

Changes in BIOS:

* Press F2 to get into BIOS Setup, or F12 and then select BIOS Setup.
* Disable "Secure Boot" (under "Boot Configuration")
* Change Fastboot behavior to "Thorough" (under "Pre-boot Behavior" tab)
* No BIOS settings found for USB Boot Support, but I guess this has been enabled by default. No influence on the following installation steps.
* Save and Exit.

Change from RAID to AHCI (NVME SSD also needs to do this)

* Get into Windows system, run Command Prompt as Administrator
* Type command `bcdedit /set safeboot minimal` (the {current} variable doesn't work for me, removing it works fine); this will force the next Windows boot into safe mode
* Restart and get into BIOS settings, find SATA/NVME Operation (under "Storage" tab) and change from RAID to AHCI
* Save and reboot and get into Windows, run Command Prompt as Administrator, and type command `bcdedit /deletevalue safeboot` to disable safe mode boot.
* Restart and get into Windows in normal mode, make sure everything is ok here.
* Note: if you don't do these commands for safe mode boot, the system will get into blue screen and never be able to boot up. If you don't change this RAID settings, the USB installation drive won't be able to find unallocated space on the disk to install Ubuntu.

Install Ubuntu

* Prepare a USB drive for Ubuntu installation
* Plug in the USB drive, boot and get into BIOS settings, find "Boot Sequence" under "Boot Configuration", move this USB drive to the first one in boot sequence.
* Restart and the system will boot into this USB drive, and then start installation as normal.
* Better to select "Install third-party software for graphics and Wi-Fi hardware"

Upgrade Linux Kernel on Ubuntu 20 (to use WiFi on XPS 13)

```bash
# install 5.19.13
sudo add-apt-repository ppa:cappelikan/ppa
sudo apt update
sudo apt install mainline
sudo mainline install 5.19.13
# fix libc6 and libssl3 errors (or ignore them)
sudo apt update
sudo apt upgrade
sudo apt install --reinstall linux-firmware
sudo apt install --install-recommends linux-generic-hwe-20.04
sudo apt install --install-recommends linux-generic-hwe-20.04-edge
# finish installation
sudo update-grub
sudo reboot
uname -a
```

The default kernel version on Ubuntu 20 is 5.15, which does not support this XPS wifi hardware yet. We need to use the default kernel version on Ubuntu 22 (5.19.13) in order to use WiFi on this XPS.&#x20;


# Linux


# Shell

### SSH

* `ssh` to see if ssh is installed (openssh-client is installed by default)
* `ssh-keygen` or `ssh-keygen -t rsa` generate ssh key by RSA algorithm
* `cat ~/.ssh/id_rsa.pub` print public key (for settings on Github)
* `ssh -T git@github.com` testing
* `sudo apt-get install openssh-server` install server for incoming ssh connections
* `ssh-copy-id username@hostname` copy your ssh key to remote computer to avoid verifying password on every connection
* `ssh username@ip` ssh login to a remote computer with this ip and username
* `ssh username@hostname` if mapped this hostname to its ip in /etc/hosts file
* `ssh -X username@hostname` login with GUI support
* `ssh -p 1234 username@hostname` login via port 1234
* `sudo gedit /etc/hosts` modify hosts file
* `sudo gedit /etc/ssh/sshd_config` modify ssh server configuration

### Devices and Filesystem

* `sudo gparted`
* `ls -l /dev/ttyS0`
* `groups <username>`
* `sudo gpasswd -a <username> <groupname>`
* `sudo gpasswd -d <username> <groupname>`
* `sudo groupadd <newgroupname>`
* `sudo usermod -a -G <groupname> <username>`
* `lscpu` , `lsmem` , `lshw` list CPU, memory, hardware info
* `lsusb` list USB devices
* `lsblk` list block devices, often used to check external ssd
* `lspci` list PCI devices
* `lspci | grep VGA` check GPUs
* `sudo lshw -C video` check GPU display details
* `du -h -d 1` check file size/usage in current directory
* `df -h` check hard disk usage

### Environment Variables

* `which pip` to see which `pip` command you are running - useful for checking environment!
* uncomment `#force_color_prompt=yes` in `.bashrc` for a colored prompt
* `sudo !!` where `!!` is for the last command
* `python3 -m site` show searching path for python3
* `sudo python -m pip install --force-reinstall pip==8.1.1` if you [inadvertently upgraded your system pip](https://stackoverflow.com/questions/49836676/error-after-upgrading-pip-cannot-import-name-main). More info can be found at [this link](https://stackoverflow.com/questions/50776299/how-to-downgrade-pip-version-10-0-0-to-pip-version-9-0-1), and [this link](https://github.com/pypa/pip/issues/5221#issuecomment-382069604).
* `sudo ldconfig` after `sudo make install` to refresh the cache

### Date and Time

* `date`
* `sudo systemctl stop systemd-timesyncd`
* `sudo systemctl start systemd-timesyncd`
* `timedatectl`
* `sudo timedatectl set-ntp false`
* `sudo timedatectl set-time 'YYYY-MM-DD'`
* `sudo timedatectl set-time 'YYYY-MM-DD HH:MM:SS'`

### Network

* `sudo systemctl restart network-manager`
* `nmap -sP 192.168.0.1/24` scan ports
* `nmap -A 192.168.10.112` detect OS and services

### Compression

* `tar -cf filename.tar file1 folder2` packed without compression
* `tar -xf filename.tar`  create an archive by `-c`, extract an archive by `-x`
* `tar -cvzf filename.tar.gz file1 folder2` compress by `-z`; verbose by `-v`
* `tar -xvzf filename.tar.gz`  decompress
* `tar -xvzf filename.tar.gz -C {path}` decompress to a designated path
* `sudo apt-get install zip unzip`
* `zip -r archive_name.zip myfolder`
* `unzip archive_name.zip`

### running in the background

* `Ctrl + Z` to stop the process, followed by `bg` to run in the background or `fg` to run in the foreground.
* Use command `./test.sh 1>stdout.log 2>stderr.log &` and it will return a process ID.&#x20;
* Processes running in the background can be monitored by `top` command.&#x20;

### apt commands to find dependencies ([link](https://askubuntu.com/questions/13296/how-do-i-find-the-reverse-dependency-of-a-package))

* `apt-cache showpkg <pkgname>` show package info, both depends and reverse depends
* `apt-cache depends ros-melodic-desktop` show one-level dependencies
* `apt-cache rdepends --installed ros-melodic-desktop`&#x20;
* `sudo apt install apt-rdepends`
* `apt-rdepends ros-melodic-desktop | pager` recursively show all dependencies&#x20;
* `apt-rdepends --reverse protobuf-compiler | grep ros-melodic-desktop` &#x20;

### apt commands to locate packages

* `dpkg -L <package-name>`
* `apt list <package-name>`
* `apt-cache madison` &#x20;
* `locate <filename>` find a file in database (globally)
* `find <filename>` find a file in local directory

### vim commands

* `vi -b <filename>` open file in binary mode
* `:%!xxd` switch to hex mode
* `:%!xxd -r` switch back to binary mode
* `:wq` save and exit

### terminal

* `sudo update-alternatives --config x-terminal-emulator`

### References

* [The Unix Shell](http://swcarpentry.github.io/shell-novice/)
* [How do I find the reverse dependency of a package?](https://askubuntu.com/questions/13296/how-do-i-find-the-reverse-dependency-of-a-package)


# GDB

## gdb

* make sure debug flag is enabled when compiling the program.&#x20;
  * e.g., `gcc -g`&#x20;
  * e.g., `set(CMAKE_BUILD_TYPE RelWithDebInfo)` or `cmake -DCMAKE_BUILD_TYPE=Debug`&#x20;
* `gdb <program_name>`
* `gdb --args <program_name> arg1 arg2 arg3`
* `(gdb) break main` add a breakpoint at `main` function
* `(gdb) r` run the program
* `(gdb) list` list current function (in its original programming language)
* `(gdb) tbreak 48` add a temporary breakpoint at line 48 of current function
* `(gdb) c` continue running the program
* `(gdb) bt` print backtrace (of current stack)
* `(gdb) print <var>` print a variable
* `(gdb) info locals` print all local variables (of current stack)
* `(gdb) x /x 0x7fffffffd9b0` exam the memory at this address using hex format
* `(gdb) q` quit

More useful commands

* `(gdb) s` step into next function
* `(gdb) n` go to next instruction
* `(gdb) bt full` print full stack info including variables
* `(gdb) info args` print local `argc` and `argv` information&#x20;
* `(gdb) whatis v` check variable type
* `(gdb) disas` disassemble current function into assembly language
* `(gdb) list <filename:function_name>` list a function
* `(gdb) list <filename:line_number>` list a file around this line number
* `(gdb) tbreak 0x000055555576bf00` add a temporary breakpoint at the address (of an instruction)
* `(gdb) x /2xw 0xffff` print two-word length of the memory at this address using hex format
  * `b` byte, `h` half-word (2 bytes), `w` word (4 bytes), `g` giant word (8 bytes)
  * `x` hex, `a` pointer, `c` char, `d` integer, `u` uint, `s` string, `t` binary, `f` float, `o` octal

Fancy usage of extended gdb commands

* need to download and `source stl-views-1.0.3.gdb` ; can be added to `~/.gdbinit`&#x20;
* `(gdb) pvector <name>` print information about the STL vector class `std::vector<T>`&#x20;
* `(gdb) pmap <name>` print information about the STL map class `std::map<T,T>`&#x20;

references: [gdb cheatsheet](https://darkdust.net/files/GDB%20Cheat%20Sheet.pdf); [gdb commands](http://www.yolinux.com/TUTORIALS/GDB-Commands.html); [gdb STL support](https://sourceware.org/gdb/wiki/STLSupport); [download stl-views-1.0.3.gdb](https://sourceware.org/gdb/wiki/STLSupport?action=AttachFile\&do=view\&target=stl-views-1.0.3.gdb);&#x20;


# Git

* `git branch -vv` check for remote repo connection
* `git branch -r/-a` list remote/all branches
* `git branch -u origin/<branch>` set the upstream branch for current branch
* `git blame <file-name>` check for contributors
* `git cherry-pick <commit1> <commit2> ...` pick commits and append to current HEAD
* `git config --global user.email "email@example.com"`
* `git config --global user.name "Your Name"`
* `git config --global alias.s status` use `git s` in short for `git status`
* `git config --global push.default simple`
* `git config --global pull.ff only`
* `git config --global core.autocrlf true/false/input`
* `git config --global --unset <entry-name>` unset global configuration
* `git clone -b <branch-name> --single-branch <url>`
* `git commit --amend` re-commit the last commit
* `git checkout -` go to previous branch
* `git checkout <branch>/<commit-id>/<tag>`
* `git checkout -b <branch>` create a new branch and checkout to it
* `git checkout -b <branch> origin/<branch>` also specify the upstream branch&#x20;
* `git checkout <file-name>` undo a file to the last commit (valid before `git add` this file)
* `git clean <file-name> -f` remove untracked file
* `git clean <directory-name> -df` remove untracked directory
* `git clean -X -f` remove files ignored by `.gitignore`
* `git diff --staged/cached` diff between last commit and staged files
* `git diff <id1> <id2>`
* `git diff <branch1> <branch2>`
* `git describe --tags --abbrev=0` most recent tag in current branch
* `git log --graph --pretty=oneline --decorate --all`
* `git push --force` force update remote repo (overwrite commits in remote repo)
* `git push origin master` will push local *master* to remote *master*, regardless of current branch
* `git push origin <source>:<destination>` push local *source* branch to remote *destination* branch where `source` could also be a location represented by `HEAD~1` or `master^` &#x20;
* `git push origin :foo` push nothing to a remote branch foo will delete this branch
* `git fetch origin :foo` fetch nothing to a local branch foo will create this new branch&#x20;
* `git pull` = git fetch + git merge; two branches in history
* `git pull --rebase` = git fetch + git rebase; a single line in history (recommended)
* `git rm <file-name> -f` remove a staged file and delete it from file system
* `git rm <file-name> --cached` remove a staged file but still keep it in working directory
* `git reset <file-name>` move a file from staging to working directory
* `git reset --hard/--soft/--mixed <commit-id>`
* `git remote -v` to see the current remote URL
* `git remote set-url origin <newurl>` reset URL for origin
* `git revert <commit-id>` go back to this commit by adding a new commit
* `git reflog` see all history operations
* `git rebase <branch>`  merge into a single line by making copies of new commits
* `git rebase -i`  interaction mode (can pick commits)
* `git stash`
* `git stash list/pop/apply/drop/clear`
* `git show <branch-name>:<file-name>`
* `git status --ignored` show files ignored by `.gitignore`
* `git tag <name>`
* `git tag -ln` check for detailed info
* `git update-index --assume-unchanged <path/to/file>` untrack/hide a file
* `git update-index --no-assume-unchanged <path/to/file>` track/unhide a file

references: [Version Control with Git](http://swcarpentry.github.io/git-novice/); [Learn Git Branching](https://learngitbranching.js.org/); [How to undo almost anything with git](https://github.blog/2015-06-08-how-to-undo-almost-anything-with-git/)


# Tmux

## Motivation

At cases, we need to run a long session via SSH for heavy tasks, but the connection is not always stable and might be disconnected. If so, we are going to lose the access to these remote sessions. `tmux` can help you reconnect to a disconnected remote terminal (if you run these tasks under `tmux` from the very beginning).&#x20;

## Installation

```shell
sudo apt install tmux
sudo apt install tmuxp  # a python session manager for tmux
```

## Usage

### tmux

Outside of a session

```shell
tmux     # enable tmux in remote terminal before running time-consuming tasks
tmux a   # attach to a previous disconnected terminal launched in tmux
tmux ls  # list all terminals launched by tmux
tmux kill-session -t target-session  # kill a running session
tmux kill-session -a  # kill all sessions
tmux kill-server      # kill all sessions
```

Inside of a session

```shell
Ctrl+B, and type
  1, 2, or other numbers  # switch to window
  arrow keys  # navigate panes
  n  # go to next window
  w  # go to window mode
  t  # add tag
  x  # kill tagged window
  d  # detach current session
  :kill-session  # kill current session
  [  # enter scroll mode, press q to quit
```

To set up mouse scrolling, add the following line to `~/.tmux.conf` :

```shell
set -g mouse on   # For tmux version 2.1 and up
```

Note that this setting breaks highlighting and copying text. Use shift+click to maintain this functionality.

### tmuxp

```shell
tmuxp load config.yaml
```

* Example: <https://github.com/hanzheteng/LOCUS/blob/main/tmuxp_config/run_locus.yaml>

## Reference

* `tmux` source code: <https://github.com/tmux/tmux>
* `tmuxp` source code: <https://github.com/tmux-python/tmuxp>
* Tutorials: <https://phoenixnap.com/kb/tmux-tutorial-install-commands>
* Tmux Cheat Sheet: <https://tmuxcheatsheet.com/>&#x20;
* [How do I scroll in tmux?](https://superuser.com/a/510310/1279296)


# Network

### Static IP

![](/files/-M_cldomP-K4LvzoILZb)

### VICON Network

![](/files/-M_cllUe2jN0oEdNBLnV)


# Tricks

## Set a Python script as the default program to open `.pcd` files

With this setup, you can open/view any `.pcd` point cloud file by just double clicking the mouse.&#x20;

1. Prepare a script that takes filename as the first argument.
   * for example, running command `pcd_viewer.py example.pcd` can open the pcd file.
   * remember to add executable permission `sudo chmod +x pcd_viewer.py` .
2. Create a [MIME type](https://en.wikipedia.org/wiki/Media_type#Mime.types) for `.pcd` files.&#x20;
   * open the file `/etc/mime.types` and add a line  `application/x-pcd        pcd`,
   * such that files with extension `.pcd` can be recognized as the MIME type `application/x-pcd`.&#x20;
3. Create a `pcd-viewer.desktop` file to link/associate the program to the designated MIME type.
   * set the execution path to the prepared python script, and add `%u` to help pass the argument
   * add a line `MimeType=application/x-pcd` to declare that this app can handle this MIME type
   * place this file in `/usr/share/applications` if you plan to make it work for all users
   * place this file in `~/.local/share/applications` for only the current user
   * with this file set properly, you can see it in the "Open with" menu after right click (please verify this before moving forward; it would not show up if `%u` is not added or the path is not valid)
4. Set this application as the default program for `.pcd` files.
   * option one: right click an`.pcd` file and select PCD Viewer; it will remember the last program as the default one to open it in the future.&#x20;
   * option two: open the file `/usr/share/applications/defaults.list` and add a line `application/x-pcd=pcd-viewer.desktop` ; this will force it to use PCD Viewer at all time.
   * note: creating a `defaults.list` at `~/.local/share/applications` would not work.&#x20;
   * To make it work immediately you can run `sudo update-desktop-database` after the changes.

```python
# file: pcd_viewer.py
#!/usr/bin/env python3
import sys
import os
import open3d as o3d
if __name__ == "__main__":
    if 'SVGA_VGPU10' in os.environ:
        del os.environ['SVGA_VGPU10']
    pcd = o3d.io.read_point_cloud(sys.argv[1])
    o3d.visualization.draw_geometries([pcd])
```

```shell
# file: open3d_pcdviewer.desktop
[Desktop Entry]
Encoding=UTF-8
Version=1.0
Type=Application
Name=PCD Viewer
Icon=python3.8  # full path is /usr/share/pixmaps/python3.8.xpm
Terminal=false
Exec=/opt/pcd_viewer.py %u 
MimeType=application/x-pcd
```

Reference: [Ubuntu – Make shell script default to run file extension](https://itectec.com/ubuntu/ubuntu-make-shell-script-default-to-run-file-extension/)

## Set Snap Store applications as the default program

The same approach can be applied to applications installed from Snap Store.

* Run `snap list` to see all applications installed from Snap Store.
* Run `echo $XDG_DATA_DIRS` to check the env path for all installed applications.
* Their `.desktop` files are placed at `/var/lib/snapd/desktop/applications/` .

Just two steps needed:

1. Register .pcd as the MIME type if not done yet. (Similar to the Step 2 to set up a python script as the default program.)
2. Place the following .desktop file under `~/.local/share/applications`.

```shell
# file: cloudcompare_pcdviewer.desktop
[Desktop Entry]
Encoding=UTF-8
Version=1.0
Type=Application
Name=CloudCompare PCDViewer
Icon=/snap/cloudcompare/208/meta/gui/cloudcompare.png
Terminal=false
Exec=/snap/bin/cloudcompare.CloudCompare %u
MimeType=application/x-pcd
```

Reference: [Launchers for installed snap applications](https://askubuntu.com/a/1237852/1185093)

## Auto start a program/script when the system boots

If `sudo` privilege is required, we will need three files; otherwise two files. The usage is the following.&#x20;

1. Prepare the script you plan to run as the system starts, named `launch.sh` for example.
2. If root privilege is needed, prepare a file named `root.sh` to run the `launch.sh` file.&#x20;
3. Prepare a `launch.desktop` file to run `launch.sh` or `root.sh` script. Place this file in `~/.config/autostart/` folder.
4. Reboot the computer and you will see the program running automatically as the system starts.

```bash
# file: launch.sh
#!/bin/bash
source /opt/ros/indigo/setup.bash
source /home/ubuntu/catkin_ws/devel/setup.bash
roslaunch cruiser cruiser_manifold.launch

# file: root.sh
echo "password" | sudo -S /home/ubuntu/Desktop/launch.sh

# file: launch.desktop
[Desktop Entry]
Encoding=UTF-8
Name=LaunchROS
Type=Application
Exec=/home/ubuntu/Desktop/root.sh
Terminal=true
Categories=Application;
```

Reference: [Cruiser-OnboardROS autostart](https://github.com/CruiserProject/Cruiser-OnboardROS#notice)

## Mount USB Drive in Remote Sessions without Root Privilege

* When working on GUI desktop (with monitor, mouse and keyboard), if you plug in a USB flash drive or SD card, it will get mounted automatically and file manager will prompt out. The mounting point is at `/media/<username>/<usb-drive-id>`.
* In remote login sessions (where you only have ssh terminals), if you plug in a USB flash drive, nothing happens. You have to mount it manually by the [`mount`](http://man7.org/linux/man-pages/man8/mount.8.html)command with `sudo`.&#x20;

```
sudo mkdir /mnt/ssd
sudo mount /dev/sda1 /mnt/ssd
sudo umount /mnt/ssd
```

* The drawback of this operation is that, you have to use `sudo`for all write operations in this directory later on, because the owner is root rather than the current user. The solution is to mount it by [`udisks`](https://linux.die.net/man/1/udisks)command instead. (Actually in GUI desktop, nautilus file manager does this for you.)

```bash
udisks --mount /dev/sdb1
```

Reference: [CLI mounting vs. GUI mounting](https://askubuntu.com/questions/270775/cli-mounting-vs-gui-mounting)

## Secure Erase a Portable SSD

* For SanDisk SSD, there is an official software: [SanDisk SSD Dashboard](https://kb.sandisk.com/app/answers/detail/a_id/15108/~/dashboard-support-information)
  * This software installer requires network connection to install. (\~100MB)
  * "*Secure Erase* is different from *Sanitize* because it only deletes the mapping table but will not erase all blocks that have been written to. Sanitize will delete the mapping table and will erase all blocks that have been written to. Therefore, Secure Erase is faster to complete than Sanitize."
  * There may not be a secure erase option available for some models of SSD. (e.g., not available for my Extreme PRO 4TB)
* Secure Erase under Windows
  * For HDD
    * DBAN: Short for Darik’s Boot and Nuke, DBAN has been around for years and is a well-known and trusted drive wipe utility for HDDs. It does multiple pass rewrites (binary ones and zeros) on the disk. You’ll need to download it to a USB drive and run it from there.&#x20;
    * Disk Wipe: Disk Wipe is another free utility that does multiple rewrites of binary data. You can choose from a number of different methods for overwriting your disk. Disk Wipe is also portable, so you don’t need to install it to use it.&#x20;
    * Eraser: Eraser is also free to use. It gives you the most control over how you erase your disk. Like Disk Wipe, you can choose from different methods that include varying numbers of rewrites, or you can define your own.
  * For SSD
    * Parted Magic: Parted Magic is the most regularly recommended third-party erase tool for SSDs, but it does cost $13. It’s a bootable tool like some of the HDD erase tools—you have to download it to a USB drive and run it from there. (Note: there might be some executable available for download at 3rd party websites.)
    * ATA Secure Erase: ATA Secure Erase is a command that basically shocks your SSD. It uses a voltage spike to flush stored electrons. While this sounds damaging (and it does cause some wear), it’s perfectly safe. It doesn’t overwrite the data like other secure erase tools, so there’s actually less damage done to the SSD.
* Secure Erase under Linux
  * Option One: Using Parted Magic
    * Download a ISO image from some 3rd party software website for free.&#x20;
    * Create a new VM in VMware using this image.
    * Forward SSD to this Parted Magic VM and perform secure erase.
  * Option Two: Command-line Simple Secure Erase
    * `sudo hdparm -I /dev/sdX` Displays all information about a drive.
    * `sudo hdparm ––security-erase PASS /dev/sdX` Simple command to secure erase a drive with no hassle.
    * `sudo hdparm ––user-master u ––security-set-pass PASS /dev/sdX` Sets a password.&#x20;
    * `sudo hdparm ––user-master u ––security-erase PASS /dev/sdX` Securely erases the SSD.
  * More information for Option Two: check [this article](https://www.thomas-krenn.com/en/wiki/Perform_a_SSD_Secure_Erase).

References: [SanDisk Dashboard Support Information](https://kb.sandisk.com/app/answers/detail/a_id/15108/~/dashboard-support-information); [Getting Rid of Your PC? Here’s How to Wipe a Windows SSD or Hard Drive](https://www.backblaze.com/blog/how-to-wipe-pc-ssd-or-hard-drive/); [Securely Erasing Your SSD with Linux: A How-To](https://techgage.com/article/securely-erasing-your-ssd-with-linux-a-how-to/); [Perform a SSD Secure Erase](https://www.thomas-krenn.com/en/wiki/Perform_a_SSD_Secure_Erase)

## Increase Swap Space

Normally, it is safe to increase your swap size to match the RAM size. For example, if you RAM size is 16GB, you can increase the swap size to be 16GB as well.&#x20;

```shell
# Turn swap off
# This moves stuff in swap to the main memory and might take several minutes
sudo swapoff -a

# Create an empty swapfile
# Note that "1G" is basically just the unit and count is an integer.
# Together, they define the size. In this case 16GB.
sudo dd if=/dev/zero of=/swapfile bs=1G count=16

# Set the correct permissions
sudo chmod 0600 /swapfile

sudo mkswap /swapfile  # Set up a Linux swap area
sudo swapon /swapfile  # Turn the swap on
```

References: [AskUbuntu: How to increase swap space?](https://askubuntu.com/a/1177939/1185093); [How To Add Swap Space on Ubuntu 20.04](https://www.digitalocean.com/community/tutorials/how-to-add-swap-space-on-ubuntu-20-04)


# Debug FAQ

## Gazebo crashed when connecting to image\_view

This can happen in Ubuntu 16 and under VMware environment.

example error messages:&#x20;

> vmw\_ioctl\_command error Invalid argument

> gzserver: /build/ogre-1.9-mqY1wq/ogre-1.9-1.9.0+dfsg1/OgreMain/src/OgreRenderSystem.cpp:546: virtual void Ogre::RenderSystem::setDepthBufferFor(Ogre::RenderTarget\*): Assertion \`bAttached && "A new DepthBuffer for a RenderTarget was created, but after creation" "it says it's incompatible with that RT"' failed.

solution: [upgrade](http://gazebosim.org/tutorials?tut=install_ubuntu#Alternativeinstallation:step-by-step) to Gazebo 7 by the following commands.

```
sudo sh -c 'echo "deb http://packages.osrfoundation.org/gazebo/ubuntu-stable `lsb_release -cs` main" > /etc/apt/sources.list.d/gazebo-stable.list'
wget http://packages.osrfoundation.org/gazebo.key -O - | sudo apt-key add -
sudo apt-get update
sudo apt-get install gazebo7 -y
```

If this cannot solve the problem, run the following command to disable OpenGL 3.0 (and use OpenGL 2.1 instead), or disable "Accelerate 3D Graphics" in VM display settings. In addition, increasing the amount of graphics memory to 2GB (in VM display settings) may also help.

```
echo "export SVGA_VGPU10=0" >> ~/.bashrc
```

reference: [GitHub issue](https://github.com/uzh-rpg/rpg_quadrotor_control/issues/58#issuecomment-499814392), [Gazebo forum](https://answers.gazebosim.org//question/13214/virtual-machine-not-launching-gazebo/)

## Gazebo: Error in REST request&#x20;

This is a known issue when running Gazebo in Ubuntu 18, which can cause the model spawn service to fail. The error message is the following.

> \[Err] \[REST.cc:205] Error in REST request

Solution: open the file `gedit ~/.ignition/fuel/config.yaml` and replace the url inside from `https://api.ignitionfuel.org` to `https://api.ignitionrobotics.org`.

Reference: [Gazebo forum](https://answers.gazebosim.org/question/25030/gazebo-error-restcc205-error-in-rest-request/)

## Support exFAT File System in Ubuntu

* If you have an USB drive with 32GB or less flash memory, it is high likely using [FAT32 ](https://en.wikipedia.org/wiki/File_Allocation_Table#FAT32)file system by default. One salient drawback of this file system is that the maximum file size is 4GB. If you have a 5GB ROS bag (logging data) locally on the robot, you cannot copy it to this USB drive.&#x20;
* The good news is you can format this flash drive into [exFAT ](https://en.wikipedia.org/wiki/ExFAT)file system, which can support much larger size for a single file. However, this file system is not natively supported by Ubuntu 16.04. The solution is to install the following packages.

```bash
sudo apt update 
sudo apt install exfat-fuse exfat-utils
```

Reference: [How to Mount an exFAT Drive on Ubuntu Linux](https://linuxize.com/post/how-to-mount-an-exfat-drive-on-ubuntu/)

## A stop job is running for Session c1 of user gdm (1min 30s)

This happens to Ubuntu 18 (and above probably) Linux OS. The shutdown process is not clean and the user has to wait 1min 30s for the shutdown process to complete. By playing with `journalctl` , I was able to identify the daemon process is Xwayland.

The observed behavior is prevented if Xorg instead of Wayland is used for the login screen in gdm3. Therefore, edit section `[daemon]` in `/etc/gdm3/daemon.conf` as follows:

```
/etc/gdm3/daemon.conf:
[daemon]
# Uncomment the line below to force the login screen to use Xorg
WaylandEnable=false
```

Reference: [Stack Exchange](https://unix.stackexchange.com/questions/273876/a-stop-job-is-running-for-session-c2-of-user/496563#496563), [Debian Bug report logs](https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=843090#10)

## Shortcut Ctrl+Shift+E opens Emoji&#x20;

This can happen on an Intel NUC computer when trying to split the window vertically in `terminator`.

To change the OS' emoji picker shortcut, we can open the `ibus-setup` GUI and setting a different shortcut under the `Emoji` tab as described [here](https://askubuntu.com/questions/1039008/how-can-i-change-the-keyboard-shortcut-for-emoji-picker/1039039#1039039).

Or you can do it on the terminal as described [here](https://askubuntu.com/questions/1039008/how-can-i-change-the-keyboard-shortcut-for-emoji-picker/1141692#1141692). To completely disable the emoji picker shortcut, run:

```
gsettings set org.freedesktop.ibus.panel.emoji hotkey "@as []"
```

## (Deprecated) Debug Errors & Solutions

### **1. install pylint**

When installing `pylint` by `sudo /usr/bin/python -m pip install pylint`, the error message is

```
    Complete output from command python setup.py egg_info:
    Error: typed_ast only runs on Python 3.3 and above.

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-hZvW3z/typed-ast/
```

solution: `pip install --upgrade setuptools`

When using RoboWare, we sometimes see the prompt message to remind us of installing `pylint`. If click the install button, the command run by itself would be `sudo /usr/bin/python -m pip install pylint`. This will install `pylint` from `pip`. Then you may see the other error message

```
ERROR: Cannot uninstall 'enum34'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.
```

This is because the package `enum34` is managed by `apt` rather then `pip`. If you then try to uninstall `enum34` in pip, you will see that many (103 on my computer) ROS packages will be removed at the same time, because these ROS Kinetic packages depend on this specific version of `enum34`, and `apt` will automatically manage these dependencies for you. So we cannot just simply uninstall or upgrade this package.

Given the fact that we cannot modify `enum34`, how could we solve this problem? The solution could be a "detour" plan. That is to install `pylint` directly from `apt` rather than `pip`, though `pip` was run automatically by RoboWare.

To search which package to install: `apt search pylint`. Then we can see three possible options: `pylint`, `python-pylint-common`, and `python3-pylint-common`. The first one is good enough. Then just run `sudo apt install pylint`. Done!

Finally, check if this package was installed successfully by `pip list | grep pylint`.

### **2. python-environment-related problems**

```
AttributeError: type object 'numpy.ndarray' has no attribute '__array_function__'
```

It is high likely that you are running a wrong version python other than required. Check your environment by

* `which python`
* `which python3`
* `which pip`
* `which pip3`
* `env | grep python`
* `env | grep PATH`
* `env | grep PWD`
* `pip list | grep cflib`
* `pip show cflib`
* `pip3 show cflib`

For example: in `virtualenv` environment, running `which pip` may return `/home/username/venv/bin/pip` rather than `/usr/bin/pip`.

For the error message above, the reason is that `cfclient` is running a wrong version of `numpy` library. A related discussion can be found in [this issue](https://github.com/numpy/numpy/issues/12736#issuecomment-454442867) in `numpy` github repo.

This problem is due to the residual files of `numpy` of version `1.15.4`, which should be clearly uninstalled before installing the `1.16.0` version.

Showing solution in code,

* `pip3 uninstall numpy`
* `pip3 install numpy==1.15.4`

would solve this issue.

Also, continuing to execute the following two lines would upgrade `numpy` to `1.16.0` and `cfclient` would still work.

* `pip3 uninstall numpy`
* `pip3 install numpy`

By exploring files in `~/.local` directory, we can see that when this issue happens, there is always a `numpy-1.15.4.dist-info` directory in position `~/.local/lib/python3.5/site-packages`, no matter how you uninstall and reinstall numpy. However, after reverting to numpy of version `1.15.4` and uninstalling it, this directory goes away, and only the `numpy-1.16.0.dist-info` directory left there.

### **3. ROS error in Python script**

```
Traceback (most recent call last):
  File "./ucr_demo.py", line 72, in <module>
    cfs[i].goTo(goal=C[i], yaw=0, duration=3.0)
  File "/home/arcslab/catkin_ws/src/crazyflie_mrs/scripts/crazyflie.py", line 94, in goTo
    self.goToService(groupMask, relative, gp, yaw, rospy.Duration.from_sec(duration))
  File "/opt/ros/kinetic/lib/python2.7/dist-packages/rospy/impl/tcpros_service.py", line 435, in __call__
    return self.call(*args, **kwds)
  File "/opt/ros/kinetic/lib/python2.7/dist-packages/rospy/impl/tcpros_service.py", line 515, in call
    responses = transport.receive_once()
  File "/opt/ros/kinetic/lib/python2.7/dist-packages/rospy/impl/tcpros_base.py", line 727, in receive_once
    p.read_messages(b, msg_queue, sock)
  File "/opt/ros/kinetic/lib/python2.7/dist-packages/rospy/impl/tcpros_service.py", line 353, in read_messages
    self._read_ok_byte(b, sock)
  File "/opt/ros/kinetic/lib/python2.7/dist-packages/rospy/impl/tcpros_service.py", line 336, in _read_ok_byte
    raise ServiceException("service [%s] responded with an error: %s"%(self.resolved_name, str))
rospy.service.ServiceException: service [/cf4/go_to] responded with an error: timeout

```

I am trying to do a five robot demo with crazyflies, but some of them failed and dropped to the ground. Then I powered off those failed robots and this error emerged. This is because the server cannot communicate with those powered off robots. This error then terminated the script and I lost control of all my robots.

Possible solutions:

1. Connect to a joystick and set up an emergency button, so than you will still have control of your robots when bad things happen.
2. Instead of solution (1), we can also directly send through command line `rosservice call /cf1/emergency` to shut down the robot. Also, it is doable to have a keyboard interface that sends takeoff, land, emergency, etc.
3. Use a "try-except" structure in the Python script to handle (actually pass) these errors. Such that when error appears, the script can still continue running the remaining part.

```
    try:
        run()
    except rospy.ROSInterruptException:
        pass
```

### **4. ROS package dependency**

```
Failed to load nodelet [/camera/register_depth_rgb] of type [depth_image_proc/register] even after refreshing the cache: Failed to load library /opt/ros/kinetic/lib//libdepth_image_proc.so. Make sure that you are calling the PLUGINLIB_EXPORT_CLASS macro in the library code, and that names are consistent between this macro and your XML. Error string: Could not load library (Poco exception = libopencv_core3.so.3.3: cannot open shared object file: No such file or directory)
```

This is due to the mismatched path or name in ROS packages. The solution is to upgrade all packages to the latest. This can be solved by

* `sudo apt-get update`
* `sudo apt-get upgrade`
* `sudo apt-get dist-upgrade`


# CMake

#### References

* [CMake Documentation](https://cmake.org/cmake/help/latest/index.html) (official, latest, top-level)
* [CMake Tutorial](https://cmake.org/cmake/help/latest/guide/tutorial/index.html) (official, latest)
* [CMake Practice](https://github.com/Akagi201/learning-cmake/blob/master/docs/cmake-practice.pdf) (tutorial from users)
* [CMake 3.0.2 Documentation](https://cmake.org/cmake/help/v3.0/index.html) (top level, minimum requirement in Ubuntu 18)
  * [CMake Variables](https://cmake.org/cmake/help/v3.0/manual/cmake-variables.7.html)
  * [CMake Commands](https://cmake.org/cmake/help/v3.0/manual/cmake-commands.7.html)
  * [CMake Build System](https://cmake.org/cmake/help/v3.0/manual/cmake-buildsystem.7.html)
    * build target and library, set property
  * [CMake Packages](https://cmake.org/cmake/help/v3.0/manual/cmake-packages.7.html)
    * about find\_package(), etc.
  * [CMake Modules](https://cmake.org/cmake/help/v3.0/manual/cmake-modules.7.html)
    * [ExternalProject](https://cmake.org/cmake/help/v3.0/module/ExternalProject.html)

#### Notes

* `std::min` and `std::max`
  * `min` and `max` are defined as macros on some systems. `std::min` and `std::max` may not be used. Use `cmMinimum` and `cmMaximum` instead.
* `size_t`
  * Various implementations have differing implementation of `size_t`. When assigning the result of `.size()` on a container for example, the result should not be assigned to an `unsigned int` or similar. `std::size_t` must not be used.


# Catkin Tools

### Introduction

* Installation: `sudo apt install python-catkin-tools`
* [documentation](https://catkin-tools.readthedocs.io/en/latest/index.html), [cheat sheet](https://catkin-tools.readthedocs.io/en/latest/cheat_sheet.html)
* It's time to migrate from `catkin_make` to `catkin build` !

### `catkin build` command

* `rosbuild` was the very first build tool developed by ROS community
* Then comes `catkin_make` to help automate the merged build process at the top level
* To address the numerous drawbacks of the merged build process,`catkin_make_isolated` was introduced to isolated build process for each package
* Finally, `catkin build` is the most recent method to build packages, with improved performance
* See [catkin history](https://catkin-tools.readthedocs.io/en/latest/history.html) for more information

### catkin devel space

* In addition to the `merged` and `isolated` devel space layouts provided by `catkin_make` and `catkin_make_isolated`, respectively, `catkin_tools` provides a default [`linked`](https://catkin-tools.readthedocs.io/en/latest/advanced/linked_develspace.html) layout which enables robust cleaning of individual packages from a workspace.
* Specifically, catkin packages are built in the `.private` hidden directory at the root of the devel space. Files in devel space are symbolic links to files in `.private`folder.&#x20;
* Additionally, to avoid race condition on setup files (e.g., `setup.bash`) and other problems in parallelize building, in `linked` layout only one package generates these files. A package named `catkin_tools_prebuild` may be built first (before all other packages) for this purpose.&#x20;
* To use `merged` layout, run command `catkin config --merge-devel`.

### example catkin commands

```bash
#mkdir -p ~/catkin_ws/src
#cd ~/catkin_ws
catkin init  # cf. catkin_init_workspace

catkin config --extend /opt/ros/melodic  # extend explicitly
catkin config --merge-devel              # linked --> merged
catkin config --cmake-args -DCMAKE_BUILD_TYPE=Release

#cd src
catkin create pkg example
# cf. catkin_create_pkg example

catkin create pkg example --catkin-deps std_msgs
# cf. catkin_create_pkg example std_msgs
# deps will be added to catkin COMPONENTS

catkin create pkg example --system-deps PCL
# no corresponding command in old style (catkin_make system)
# will generate a new line: find_package(PCL REQUIRED)

catkin list  # List the packages in the workspace

catkin build  # can run at all levels (~/catkin_ws, ~/catkin_ws/src, within pkg)
catkin build example_pkg  # build only this package
cd ~/catkin_ws/src/example_pkg && catkin build --this
#echo 'source ~/catkin_ws/devel/setup.bash' >> ~/.bashrc

catkin clean
catkin clean example_pkg  # clean only this package
```

### FAQ & Troubleshooting

```
WARNING: Your current environment's CMAKE_PREFIX_PATH is different from the
cached CMAKE_PREFIX_PATH used the last time this workspace was built.

If you want to use a different CMAKE_PREFIX_PATH you should call `catkin clean`
to remove all references to the previous CMAKE_PREFIX_PATH.
```

* This warning can be ignored and it will continue to use the cached path. It often happens after new workspaces are established and setup files are sourced.&#x20;
* Alternatively, it can be resolved by explicitly extending other workspaces.&#x20;
  * `catkin config --extend /opt/ros/melodic`&#x20;
  * to revert, `catkin config --no-extend`


# CMakeLists

## CMakeLists Examples

### Minimal Example

```bash
cmake_minimum_required(VERSION 3.0.2)
project(mav_exploration)

find_package(catkin REQUIRED COMPONENTS
  roscpp
  rospy
  std_msgs
  mavros_msgs
)

catkin_package()  # from catkin (not CMake)

include_directories(
  include
  ${catkin_INCLUDE_DIRS}
)

add_executable(offboard_node src/offboard_node.cpp)
target_link_libraries(offboard_node  ${catkin_LIBRARIES})
```

### Testing

```bash
# CMakeLists.txt
enable_testing()
include_directories(${PROJECT_SOURCE_DIR}/include)
add_subdirectory(test)

# test/CMakeLists.txt
set(TEST_SOURCES test_triangulation)
foreach (test_src ${TEST_SOURCES})
    add_executable(${test_src} ${test_src}.cpp)
    target_link_libraries(${test_src} ${THIRD_PARTY_LIBS} myslam)
    add_test(${test_src} ${test_src})
endforeach (test_src)
```

### Default Template

```bash
cmake_minimum_required(VERSION 3.0.2)
project(example_pkg)

## Compile as C++11, supported in ROS Kinetic and newer
# add_compile_options(-std=c++11)

## Find catkin macros and libraries
## if COMPONENTS list like find_package(catkin REQUIRED COMPONENTS xyz)
## is used, also find other catkin packages
find_package(catkin REQUIRED COMPONENTS
  roscpp
  rospy
  std_msgs
)

## System dependencies are found with CMake's conventions
# find_package(Boost REQUIRED COMPONENTS system)


## Uncomment this if the package has a setup.py. This macro ensures
## modules and global scripts declared therein get installed
## See http://ros.org/doc/api/catkin/html/user_guide/setup_dot_py.html
# catkin_python_setup()

################################################
## Declare ROS messages, services and actions ##
################################################

## To declare and build messages, services or actions from within this
## package, follow these steps:
## * Let MSG_DEP_SET be the set of packages whose message types you use in
##   your messages/services/actions (e.g. std_msgs, actionlib_msgs, ...).
## * In the file package.xml:
##   * add a build_depend tag for "message_generation"
##   * add a build_depend and a exec_depend tag for each package in MSG_DEP_SET
##   * If MSG_DEP_SET isn't empty the following dependency has been pulled in
##     but can be declared for certainty nonetheless:
##     * add a exec_depend tag for "message_runtime"
## * In this file (CMakeLists.txt):
##   * add "message_generation" and every package in MSG_DEP_SET to
##     find_package(catkin REQUIRED COMPONENTS ...)
##   * add "message_runtime" and every package in MSG_DEP_SET to
##     catkin_package(CATKIN_DEPENDS ...)
##   * uncomment the add_*_files sections below as needed
##     and list every .msg/.srv/.action file to be processed
##   * uncomment the generate_messages entry below
##   * add every package in MSG_DEP_SET to generate_messages(DEPENDENCIES ...)

## Generate messages in the 'msg' folder
# add_message_files(
#   FILES
#   Message1.msg
#   Message2.msg
# )

## Generate services in the 'srv' folder
# add_service_files(
#   FILES
#   Service1.srv
#   Service2.srv
# )

## Generate actions in the 'action' folder
# add_action_files(
#   FILES
#   Action1.action
#   Action2.action
# )

## Generate added messages and services with any dependencies listed here
# generate_messages(
#   DEPENDENCIES
#   std_msgs
# )

################################################
## Declare ROS dynamic reconfigure parameters ##
################################################

## To declare and build dynamic reconfigure parameters within this
## package, follow these steps:
## * In the file package.xml:
##   * add a build_depend and a exec_depend tag for "dynamic_reconfigure"
## * In this file (CMakeLists.txt):
##   * add "dynamic_reconfigure" to
##     find_package(catkin REQUIRED COMPONENTS ...)
##   * uncomment the "generate_dynamic_reconfigure_options" section below
##     and list every .cfg file to be processed

## Generate dynamic reconfigure parameters in the 'cfg' folder
# generate_dynamic_reconfigure_options(
#   cfg/DynReconf1.cfg
#   cfg/DynReconf2.cfg
# )

###################################
## catkin specific configuration ##
###################################
## The catkin_package macro generates cmake config files for your package
## Declare things to be passed to dependent projects
## INCLUDE_DIRS: uncomment this if your package contains header files
## LIBRARIES: libraries you create in this project that dependent projects also need
## CATKIN_DEPENDS: catkin_packages dependent projects also need
## DEPENDS: system dependencies of this project that dependent projects also need
catkin_package(
#  INCLUDE_DIRS include
#  LIBRARIES example_pkg
#  CATKIN_DEPENDS roscpp rospy std_msgs
#  DEPENDS system_lib
)

###########
## Build ##
###########

## Specify additional locations of header files
## Your package locations should be listed before other locations
include_directories(
# include
  ${catkin_INCLUDE_DIRS}
)

## Declare a C++ library
# add_library(${PROJECT_NAME}
#   src/${PROJECT_NAME}/example_pkg.cpp
# )

## Add cmake target dependencies of the library
## as an example, code may need to be generated before libraries
## either from message generation or dynamic reconfigure
# add_dependencies(${PROJECT_NAME} ${${PROJECT_NAME}_EXPORTED_TARGETS} ${catkin_EXPORTED_TARGETS})

## Declare a C++ executable
## With catkin_make all packages are built within a single CMake context
## The recommended prefix ensures that target names across packages don't collide
# add_executable(${PROJECT_NAME}_node src/example_pkg_node.cpp)

## Rename C++ executable without prefix
## The above recommended prefix causes long target names, the following renames the
## target back to the shorter version for ease of user use
## e.g. "rosrun someones_pkg node" instead of "rosrun someones_pkg someones_pkg_node"
# set_target_properties(${PROJECT_NAME}_node PROPERTIES OUTPUT_NAME node PREFIX "")

## Add cmake target dependencies of the executable
## same as for the library above
# add_dependencies(${PROJECT_NAME}_node ${${PROJECT_NAME}_EXPORTED_TARGETS} ${catkin_EXPORTED_TARGETS})

## Specify libraries to link a library or executable target against
# target_link_libraries(${PROJECT_NAME}_node
#   ${catkin_LIBRARIES}
# )

#############
## Install ##
#############

# all install targets should use catkin DESTINATION variables
# See http://ros.org/doc/api/catkin/html/adv_user_guide/variables.html

## Mark executable scripts (Python etc.) for installation
## in contrast to setup.py, you can choose the destination
# catkin_install_python(PROGRAMS
#   scripts/my_python_script
#   DESTINATION ${CATKIN_PACKAGE_BIN_DESTINATION}
# )

## Mark executables for installation
## See http://docs.ros.org/melodic/api/catkin/html/howto/format1/building_executables.html
# install(TARGETS ${PROJECT_NAME}_node
#   RUNTIME DESTINATION ${CATKIN_PACKAGE_BIN_DESTINATION}
# )

## Mark libraries for installation
## See http://docs.ros.org/melodic/api/catkin/html/howto/format1/building_libraries.html
# install(TARGETS ${PROJECT_NAME}
#   ARCHIVE DESTINATION ${CATKIN_PACKAGE_LIB_DESTINATION}
#   LIBRARY DESTINATION ${CATKIN_PACKAGE_LIB_DESTINATION}
#   RUNTIME DESTINATION ${CATKIN_GLOBAL_BIN_DESTINATION}
# )

## Mark cpp header files for installation
# install(DIRECTORY include/${PROJECT_NAME}/
#   DESTINATION ${CATKIN_PACKAGE_INCLUDE_DESTINATION}
#   FILES_MATCHING PATTERN "*.h"
#   PATTERN ".svn" EXCLUDE
# )

## Mark other files for installation (e.g. launch and bag files, etc.)
# install(FILES
#   # myfile1
#   # myfile2
#   DESTINATION ${CATKIN_PACKAGE_SHARE_DESTINATION}
# )

#############
## Testing ##
#############

## Add gtest based cpp test target and link libraries
# catkin_add_gtest(${PROJECT_NAME}-test test/test_example_pkg.cpp)
# if(TARGET ${PROJECT_NAME}-test)
#   target_link_libraries(${PROJECT_NAME}-test ${PROJECT_NAME})
# endif()

## Add folders to be run by python nosetests
# catkin_add_nosetests(test)

```

### Curses-based User Interface for CMake (ccmake)

This package provides the CMake curses interface. Project configuration settings may be specified interactively through this GUI. Brief instructions are provided at the bottom of the terminal when the program is running. The main executable file for this GUI is "ccmake".

```bash
# to install
sudo apt install cmake-curses-gui

# to use
ccmake [<options>] {<path-to-source> | <path-to-existing-build>}
```


# CMake Variables

## Variables that Provide Information

* `CMAKE_CURRENT_SOURCE_DIR`  The path to the source directory currently being processed.
* `CMAKE_CURRENT_BINARY_DIR`  The path to the binary directory currently being processed.
* `CMAKE_SOURCE_DIR`  The path to the top level of the source tree.
* `CMAKE_BINARY_DIR`  The path to the binary directory currently being processed.
  * for an in-source build, this would be the same as `CMAKE_SOURCE_DIR`
  * for an out-of-source build, this would be at `./build` folder
* `PROJECT_SOURCE_DIR`  Top level source directory for the current project.
* `PROJECT_BINARY_DIR`  Full path to build directory for project.
* `<PROJECT-NAME>_SOURCE_DIR`  Top level source directory for the named project.
* `<PROJECT-NAME>_BINARY_DIR`  Top level binary directory for the named project.
* `PROJECT_VERSION` Value given to the VERSION option of the most recent call to the `project()` command, if any.
* `CMAKE_VERSION`&#x20;
* `CMAKE_COMMAND`  The full path to the cmake executable.
* `CMAKE_GENERATOR`  The generator used to build the project.

```bash
# ~/catkin_ws/src/example_pkg
message(${CMAKE_CURRENT_SOURCE_DIR})
message(${CMAKE_SOURCE_DIR})
message(${PROJECT_SOURCE_DIR})

# ~/catkin_ws/build/example_pkg
message(${CMAKE_CURRENT_BINARY_DIR})
message(${CMAKE_BINARY_DIR})
message(${PROJECT_BINARY_DIR})
```

## Variables that Change Behavior

* `CMAKE_BUILD_TYPE`  Specifies the build type on single-configuration generators.
  * Can be set to `Debug`, `Release`, `RelWithDebInfo`, `MinSizeRel`, ...
* `CMAKE_CONFIGURATION_TYPES` Specifies the available build types on multi-config generators.
* `CMAKE_MODULE_PATH` List of directories to search for CMake modules.
  * Commands like `include()` and `find_package()` search for files in directories listed by this variable before checking the default modules that come with CMake.
* `CMAKE_INSTALL_PREFIX` Install directory used by install.
  * If “make install” is invoked or INSTALL is built, this directory is **prepended** onto all install directories.&#x20;
  * This variable defaults to `/usr/local` on UNIX and `c:/Program Files` on Windows.

#### CMAKE\_XXX\_PATH

By default it is empty, it is intended to be set by the project.

* `CMAKE_PREFIX_PATH`  Path used for searching by FIND\_XXX(), with appropriate suffixes added.
  * a.k.a. `CATKIN_DEVEL_PREFIX` in earlier CMake versions; set to `catkin_ws/devel` for example.
  * It contains the “base” directories, the FIND\_XXX() commands append appropriate subdirectories to the base directories.&#x20;
    * FIND\_PROGRAM() adds `/bin` to each of the directories in the path,&#x20;
    * FIND\_LIBRARY() appends `/lib` to each of the directories, and&#x20;
    * FIND\_FILE() and FIND\_PATH() append `/include`.&#x20;
* `CMAKE_IGNORE_PATH` Path to be ignored by FIND\_XXX() commands.
  * This is useful in cross-compiled environments where some system directories contain incompatible but possibly linkable libraries.&#x20;
* `CMAKE_INCLUDE_PATH` Path used for searching by FIND\_FILE() and FIND\_PATH().
* `CMAKE_LIBRARY_PATH` Path used for searching by FIND\_LIBRARY().
* `CMAKE_PROGRAM_PATH` Path used for searching by FIND\_PROGRAM().

#### CMAKE\_SYSTEM\_XXX\_PATH

By default it contains the standard directories for the current system. It is NOT intended to be modified by the project.

*

```
`CMAKE_SYSTEM_IGNORE_PATH`
```

* `CMAKE_SYSTEM_INCLUDE_PATH`
* `CMAKE_SYSTEM_LIBRARY_PATH`
* `CMAKE_SYSTEM_PREFIX_PATH`
* `CMAKE_SYSTEM_PROGRAM_PATH`

```bash
message("${CMAKE_BUILD_TYPE}")  # add "" to prevent error from empty variable
message("${CMAKE_CONFIGURATION_TYPES}")
message("${CMAKE_MODULE_PATH}")
message("${CMAKE_INSTALL_PREFIX}")
```

## Variables that Describe the System

* `CMAKE_HOST_SYSTEM`
* `CMAKE_SYSTEM_NAME`&#x20;
* `UNIX` or `CMAKE_HOST_UNIX`
* `WIN32` or `CMAKE_HOST_WIN32`
* `MSVC`, `MSVC_VERSION`
* `ENV`: Access environment variables.&#x20;
  * Use the syntax `$ENV{VAR}` to read environment variable VAR.&#x20;
  * See also the `set()` command to set `ENV{VAR}`.

## Variables that Control the Build

* `EXECUTABLE_OUTPUT_PATH`
* `LIBRARY_OUTPUT_PATH`
* `CMAKE_RUNTIME_OUTPUT_DIRECTORY` Where to put all the RUNTIME targets when built.
  * This variable is used to initialize the RUNTIME\_OUTPUT\_DIRECTORY property on all the targets.
  * See that target property for additional information.

## Variables for Languages

* `CMAKE_<LANG>_COMPILER` The full path to the compiler for LANG.
  * `CMAKE_CXX_COMPILER`, `CMAKE_C_COMPILER`
* `CMAKE_<LANG>_FLAGS`  Flags for all build types.
  * `CMAKE_CXX_FLAGS`, `CMAKE_C_FLAGS`
  * `CMAKE_<LANG>_FLAGS_RELEASE`  Flags for Release build type or configuration.
* `CMAKE_CXX_STANDARD` New in version 3.1. Default value for CXX\_STANDARD property of targets.
* `CMAKE_CUDA_STANDARD`New in version 3.8.  &#x20;Default value for CUDA\_STANDARD property of targets.


# CMake Commands

## CMake Commands

Starting from v3.0, CMake commands are all in lower case SET() --> set()

#### Common

* `message([<mode>] "message to display" ...)`
* `set(<variable> <value> [[CACHE <type> <docstring> [FORCE]] | PARENT_SCOPE])`
* `unset(<variable> [CACHE | PARENT_SCOPE])`
* `option(<option_variable> "help string describing option" [initial value])`
* `export(EXPORT <export-name> [NAMESPACE <namespace>] [FILE <filename>])`
* `install(TARGETS targets... [EXPORT <export-name>] [...])`
* `file(WRITE filename "message to write"... )` also READ/APPEND/...
* `list(APPEND <list> [<element> ...])` also LENGTH/FIND/INSERT/REMOVE/SORT...
* `string(REGEX REPLACE <regular_expression> <replace_expression> <output variable>` `<input> [<input>...])`also FIND/COMPARE/LENGTH/TOLOWER...

```bash
message("words to user")
message(STATUS ${PROJECT_SOURCE_DIR})  # may be ignored by catkin build
# mode: (none), STATUS, WARNING, AUTHOR_WARNING, SEND_ERROR, FATAL_ERROR, DEPRECATION

set(CMAKE_BUILD_TYPE Release)  # Debug RelWithDebInfo MinSizeRel
set(LIBRARY_OUTPUT_PATH ${PROJECT_BINARY_DIR}/lib)

unset(ENV{LD_LIBRARY_PATH}) # <variable> can be an environment variable

option(BUILD_DOC "Build documentation" ON) 
# can be overwritten by "cmake -DBUILD_DOC=OFF .."

export(TARGETS ${TEASERPP_EXPORTED_TARGETS} FILE teaserpp-exports.cmake)

install(FILES cmake/teaserppConfig.cmake
        DESTINATION ${CMAKE_INSTALL_LIBDIR}/cmake/teaserpp)

file(STRINGS myfile.txt myfile)
# stores a list in the variable “myfile” 
# in which each item is a line from the input file.

list(APPEND CMAKE_MODULE_PATH ${PROJECT_SOURCE_DIR}/cmake)

string(REGEX REPLACE "/$" "" CMAKE_INSTALL_PREFIX "${CMAKE_INSTALL_PREFIX}")
```

#### Flow Control

* `if()`, `elseif()`, `else()`, `endif()`
* `while()`, `endwhile()`
* `foreach()`, `endforeach()`
* `macro()`, `endmacro()`
* `function()`, `endfunction()`
* `break()`, `return()`
* `enable_language(<lang> [OPTIONAL] )`
* `enable_testing()`

#### Build Target

* `find_file (<VAR> name1 [path1 path2 ...])`
* `find_library (<VAR> name1 [path1 path2 ...])`
* `find_package(<package> [version] [EXACT] [QUIET] [MODULE] [REQUIRED] [[COMPONENTS]` `[components...]] [OPTIONAL_COMPONENTS components...] [NO_POLICY_SCOPE])`
* `find_path (<VAR> name1 [path1 path2 ...])`
* `find_program (<VAR> name1 [path1 path2 ...])`
* `add_compile_options(<option> ...)`
* `add_definitions(-DFOO -DBAR ...)`: Adds -D define flags to the compilation of source files.
* `add_dependencies(<target> [<target-dependency>]...)`
* `add_executable(<name> source1 [source2 ...])`
* `add_library(<name> [STATIC | SHARED | MODULE] source1 [source2 ...])`
* `add_subdirectory()`
* `add_test()`
* `include_directories()` for all build targets
* `target_include_directories()` for a specific build target
* `target_link_libraries()` link to a specific target
  * Header-only libraries (e.g., Eigen3) are not needed to be linked here.
  * `target_link_directories()` is not encouraged to use

#### Build Dependencies

* `configure_file(<input> <output> [...])` Copy a file to another location and modify its contents.
* `execute_process(COMMAND <cmd1> [args1...]] [WORKING_DIRECTORY <directory>] [...])`&#x20;
* `include(<file|module>[...])` Load and run CMake code from a file or module.

```bash
cmake_minimum_required(VERSION 3.10)

project(tinyply-download NONE)

include(ExternalProject)
ExternalProject_Add(pmc
        GIT_REPOSITORY    https://github.com/jingnanshi/tinyply.git
        GIT_TAG           0b9fff8e8bd4d37256554fe40cf76b2f3134377b
        SOURCE_DIR        "${CMAKE_CURRENT_BINARY_DIR}/tinyply-src"
        BINARY_DIR        "${CMAKE_CURRENT_BINARY_DIR}/tinyply-build"
        CONFIGURE_COMMAND ""
        BUILD_COMMAND     ""
        INSTALL_COMMAND   ""
        TEST_COMMAND      ""
        )
```


# CMake: find\_package()

## What You Should Know & Your Actions

When we need to add an external project as the dependency, we will use command `find_package` in `CMakeLists.txt` to let CMake know where the header files and libraries are located, such that they can be found and linked properly later on.&#x20;

It is ideal that the developer of the external project provides a `<package>Config.cmake` file (or a `<lower-case-name>-config.cmake` file) for us/users, and this file will be cached into database (in the default installation path) after we `sudo make install` this external project. You can check if they are available by the `locate` command.&#x20;

```bash
# example commands
locate OpenCVConfig.cmake
locate gflags-config.cmake

# the results (if available)
/usr/share/OpenCV/OpenCVConfig.cmake
/usr/lib/x86_64-linux-gnu/cmake/gflags/gflags-config.cmake
```

If they are available, you only need to write one single line in the `CMakeLists.txt` file. For example,&#x20;

```bash
find_package(PCL)   # it will work given that PCLConfig.cmake is available
```

Unfortunately, there are certain packages that do not provide (they should!) this CMake configuration file (or the files are not adopted by Ubuntu/apt). In this case, we need to use `Find<package>.cmake` file instead to locate this external project. We can write it on our own, but it is often the case that someone has done it before. So go ahead and find an available one to use.&#x20;

After you find a good `Find<package>.cmake` file, we often place it under a folder named `cmake` or `cmake-modules` in the root directory of the current project (in parallel to `include`, `src`, etc.), and then add the path to this folder to a CMake variable `CMAKE_MODULE_PATH` (which is empty by default). With this setup, CMake will be able to look for your customized `Find<package>.cmake` file according to this CMake module path. What you need to do in `CMakeLists.txt` file is the following two lines of code.&#x20;

```bash
list(APPEND CMAKE_MODULE_PATH "${PROJECT_SOURCE_DIR}/cmake-modules")
find_package(Glog) # works given that FindGlog.cmake is placed under cmake-modules
```

#### Summary

* The CMake command `find_package` has two modes (Config mode and Module mode) to find and load the CMake configuration of an external project.
* In Config mode, it will look for a `<package>Config.cmake` or`<lower-case-name>-config.cmake` file in the default install path in Ubuntu system.&#x20;
* In Module mode, it will look for a `Find<package>.cmake` file in the path specified by variable `CMAKE_MODULE_PATH` ; this variable is empty unless set by user.
* Module mode has higher priority than Config mode, which means that you can overwrite the default Config file by providing your own FindXXX file.&#x20;

## More Technical Details

According to [the official documentation](https://cmake.org/cmake/help/v3.0/command/find_package.html) and [this stackoverflow answer](https://stackoverflow.com/questions/20746936/what-use-is-find-package-if-you-need-to-specify-cmake-module-path-anyway), find\_package() has two modes: "Module" mode and "Config" mode. If no module is found and the MODULE option is not given, the command proceeds to Config mode.

### Module mode

* It will look for a file called `Find<package>.cmake`, first in CMAKE\_MODULE\_PATH (empty by default), and then under cmake installation path `<CMAKE_ROOT>/Modules` (e.g., `/usr/share/cmake-3.5/Modules`). Some example default modules include `FindCUDA.cmake` or `FindOpenGL.cmake`.&#x20;
* Therefore, to add our own cmake modules or overwrite system cmake modules, we can add `Find<package>.cmake` files in a folder called `cmake` and do the following in CMakeLists.txt, to make the best use of CMAKE\_MODULE\_PATH.
* ```
  list(APPEND CMAKE_MODULE_PATH "${PROJECT_SOURCE_DIR}/cmake")
  ```

### Config mode

* It will look for a file called `<package>Config.cmake` or `<lower-case-name>-config.cmake` under CMAKE\_INSTALL\_PREFIX (default to `/usr/local`). These cmake files are installed when we do `sudo make install` for 3rd party libraries. Examples are as follows (excerpts from command output).&#x20;
* ```
  -- Installing: /usr/local/share/OpenCV/OpenCVConfig.cmake
  -- Up-to-date: /usr/local/share/sophus/cmake/SophusConfig.cmake
  -- Up-to-date: /usr/local/lib/cmake/Ceres/CeresConfig.cmake
  -- Up-to-date: /usr/local/lib/cmake/g2o/g2oConfig.cmake
  -- Up-to-date: /usr/local/lib/cmake/GTest/GTestConfig.cmake
  -- Up-to-date: /usr/local/lib/cmake/DBoW3/DBoW3Config.cmake
  -- Up-to-date: /usr/local/lib/cmake/Pangolin/PangolinConfig.cmake
  ```

### References

* [stackoverflow: two modes in find\_package](https://stackoverflow.com/a/20857070/13980439)
* [stackoverflow: could not find a package configuration error](https://stackoverflow.com/a/65046283/13980439)

## Commands

Commonly used commands to figure out where packages are installed.

* `dpkg -L <package-name>`
* `apt list <package-name>`
* `locate <package>Config.cmake`

To set flags

* `set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=c++11")`&#x20;
* `set(CMAKE_BUILD_TYPE RelWithDebInfo)`&#x20;
* `cmake -DCMAKE_BUILD_TYPE=Debug`

Note that all names marked by \<xxx> should be replaced by their actual file names or package names. Case sensitive!

## Actions Needed After A Package Is Found

In general, once a package is found, a new variable `<package>_FOUND` will be generated and set to true. Also, there will be a few more variables loaded into the compilation process, such as `<package>_INCLUDE_DIR` , `<package>_INCLUDE_DIRS`, `<package>_LIBRARIES`, `<package>_LIBS`. Then you can use them to let CMake know where to look for header files and libraries. For example,&#x20;

```bash
find_package(Glog REQUIRED) # REQUIRED: will stop compilation if package is missing
include_directories(${GLOG_INCLUDE_DIRS})

add_executable (main src/main.cpp)
target_link_libraries (main ${GLOG_LIBRARIES})
```

Note that different projects may adopt different variable names. You need to make sure the spelling is completely correct. (Should it be `LIBRARIES` or `LIBS`?)

Typically, the variable names are expected to be documented in the official documentation of this external project. (They should!) If not, you will need to dig into `<lower-case-name>-config.cmake`  or `<package>Config.cmake` or `Find<package>.cmake` files, which is where they are defined.


# ROS


# Gazebo

## Download Gazebo models from official database

```bash
#!/bin/sh

# Download all model archive files
wget -l 2 -nc -r "http://models.gazebosim.org/" --accept gz

# This is the folder into which wget downloads the model archives
cd "models.gazebosim.org"

# Extract all model archives
for i in *
do
  tar -zvxf "$i/model.tar.gz"
done

# Copy extracted files to the local model folder
cp -vfR * "$HOME/.gazebo/models/"
```

## Bug Fix: Error in REST request

This is a known bug in Ubuntu 18 and above.&#x20;

* launch Gazebo simulator for the first time, so that the `~/.ignition` folder will be created.&#x20;
* open file `~/.ignition/fuel/config.yaml` and replace the url inside from `https://api.ignitionfuel.org` to `https://api.ignitionrobotics.org`.

## Bug Fix: vmw\_ioctl\_command error Invalid argument

This can happen (and only happens) in virtual machine environments.

* disable OpenGL 3.0 and use OpenGL 2.1 by enabling the environment variable `SVGA_VGPU10`
* run `echo "export SVGA_VGPU10=0" >> ~/.bashrc` command **once** and restart terminals

Remember to unset this environment variable if you need OpenGL for graphics (e.g., point cloud visualization in Open3D).


# wstool

## wstool: ROS Workspace Tool&#x20;

```bash
# install wstool
sudo apt-get install python-wstool

# create a workspace
mkdir -p ~/catkin_ws/src
cd ~/catkin_ws
catkin init
#catkin config --merge-devel
#echo 'source ~/catkin_ws/devel/setup.bash' >> ~/.bashrc

# approach one: in src level
cd ~/catkin_ws/src
git clone <ros_package>.git
wstool init
#wstool init . <ros_package>/ssh.rosinstall
wstool merge <ros_package>/ssh.rosinstall
wstool update

# approach two: in workspace level
cd ~/catkin_ws
wstool init src   # will create an empty .rosinstall file 
#wstool init src PATH_TO_ROSINSTALL_FILE.rosinstall  # initialize with a .rosinstall file
wstool merge -t src PATH_TO_ROSINSTALL_FILE.rosinstall
wstool update -t src   # -t specify targer folder

# example: use rosinstall_generator
cd ~/catkin_ws
wstool init src
rosinstall_generator --rosdistro kinetic mavlink | tee /tmp/mavros.rosinstall
rosinstall_generator --upstream mavros | tee -a /tmp/mavros.rosinstall # -a: append
wstool merge -t src /tmp/mavros.rosinstall
wstool update -t src -j4
rosdep install --from-paths src --ignore-src -y

# compile code and refresh workspace
catkin build
source ~/catkin_ws/devel/setup.bash
```

#### references

* [wstool ROS Wiki](http://wiki.ros.org/wstool)
* [rosinstall\_generator ROS Wiki](http://wiki.ros.org/rosinstall_generator)
* [mavros source installation](https://github.com/mavlink/mavros/tree/master/mavros#source-installation)


# roslaunch

This note is trying to answer the following two questions.

* What happens after I type `roslaunch` in the terminal?
* How could the Linux know the commands and help me auto-complete it?

We will introduce some basics about environment variables in Linux and the concept of search path.&#x20;

### Environment Variables in Linux

![Three important PATH env in ROS](/files/-M1dI_O5-keN2nnaCWcd)

First of all, no matter what kind of command you type in the terminal, it must exist in one of the directories indicated in the **PATH** environment variable. This is where the Linux terminal will search first. For example, `ls` `cat` commands are in `/bin` directory;  `python` command is under directory`/usr/bin`.&#x20;

To auto-complete, the system must know all possible commands given the existing characters you type in the terminal. This is somehow like to complete a English word given the first X characters, where you can search in a dictionary for all possible solutions. Likewise, Linux can search all executables in the directories listed in PATH env. Once it finds there is a **unique** solution, it will help you complete it.

To find the exact path for each command you type in, you can use `which` command. For example, if you try`which python` in the terminal, it will return to you the search result of the command `python`. You can also do this for the `which` command itself. Try it.

```
user@hostname$ which python
/usr/bin/python
```

### Where ROS commands are installed?

Next, we will talk about how ROS handles the search path.

For all ROS-related commands, in order to run in the terminal, it has to be an Linux executable. This is an universal rule in Linux with no exceptions. Note that the first directory listed in PATH env is `/opt/ros/kinetic/bin`. You got it! This is where all the ROS commands exist. Open the folder and you will find python scripts like `roslaunch`, `rqt_graph`, etc.

Go ahead and open the `roslaunch` script. What? Only two lines of code?&#x20;

```
#!/usr/bin/python

# Software License
# etc...

import roslaunch
roslaunch.main()
```

Yes, it is. `roslaunch` was installed as a Python 2.7 library in the system.&#x20;

Then the following question is, how could python know where to import `roslaunch`?

Check the first directory in the **PYTHONPATH** env!

`/home/hteng/catkin_ws/devel/lib/python2.7/dist-packages`

Here we go! Open this folder and you will see all the ROS-related python programs.

### How could ROS auto-complete the package name?

The third environment variable we would like to talk about is the **ROS\_PACKAGE\_PATH**. This is where ROS will search when you are typing a package name right after `roslaunch` and `rosrun`.&#x20;

```
roslaunch turtlebot_br    [press Tab, then it becomes]
roslaunch turtlebot_bringup
```

How could ROS know all the possible package names? I believe now you can guess the answer. Yes, it will search all the directories listed in the **ROS\_PACKAGE\_PATH**.

That's all for today! Have fun!


# rosbag

## ROS Bag Buffer Issue

![](/files/-LyjfU1j068nOTVBi096)

In short, the problem is that the current data collection speed is greater than the maximum memory access bandwidth, such that more and more data are queued in the buffer and eventually results in queue overflow. In other words, `rosbag` is collecting too much data in a short time, such that the operating system cannot write them all in the memory/disk.

There is no actual solution to this issue, as you may always want to record more data at the same time. Some walkarounds are as follows.&#x20;

* On the recording side, you can 1) reduce the image resolution, 2) reduce the frame rate, or 3) reduce the number of topics to be recorded at the same time. The main objective is to reduce the amount of data written to the memory/disk.
* On the memory side, if the system is working on a removable disk (e.g., SD card), you can try to replace it with a faster one. If it is on your native laptop, as you know, it is hard to replace the disk. (If you can, replacing the traditional disk with SSD would help.)
* On the buffer side, you may use command `rosbag record -b 2048 <topic name>` to extend the buffer size. This can postpone the queue overflow (where the warning happens), and give you a bit more time to collect data.
* If using the above command, please note that you have to wait for a while after you terminate the `rosbag` command, and make sure the `xxx.bag.active` file has been transformed into `xxx.bag` file. (Some threads in the backend are still working on the rest of data in the buffer.)

References: [The rosbag record problem](https://github.com/stereolabs/zed-ros-wrapper/issues/345), [rosbag wiki](http://wiki.ros.org/rosbag/Commandline#rosbag_record)

## Check ROS Bag Health Condition

In the above example, one may ask "What would happen if this warning prompted?" or "Can I just use this bag file given that some messages are dropped?".

To check if a bag file is healthy or not, there is a useful command `rosbag info xxx.bag`. In the following, let's see two examples from [EuRoC MAV Dataset](https://projects.asl.ethz.ch/datasets/doku.php?id=kmavvisualinertialdatasets).&#x20;

An example of healthy ROS bag looks like the following.

![an example of healthy ROS bag](/files/-Lyk0arNMZbmSwYxTsy8)

How to tell if it is healthy or not? You can just check if the number of messages is reasonable or not. The total record time is 113 sec, and `/fcu/imu` topic has 1140 messages. It means that this topic is running at roughly 10Hz. Similarly, we can infer that the camera frame rate is 20Hz, the raw imu is running at 200Hz, and vicon is running at 100Hz.&#x20;

An example of unhealthy ROS bag could be the following.&#x20;

![an example of unhealthy ROS bag](/files/-Lyk0arHrmCt6asmYtAr)

In this case, the total record time is 116 sec. Therefore, given the frame rate of 20Hz, we expect that the `image_raw` topic should have about 2320 messages. However, the first camera `cam0` only has 1922 messages. It lost 300 messages during the data collection process, perhaps due to the similar buffer issue. This may affect the performance of the algorithms running on the dataset in the future.&#x20;

References: [EuRoC MAV Dataset](https://projects.asl.ethz.ch/datasets/doku.php?id=kmavvisualinertialdatasets), [rosbag wiki](http://wiki.ros.org/rosbag/Commandline#rosbag_info)


# multi-threaded spinner

### Application Scenario

* Suppose that we have two callbacks (talker1 and talker2) running at 1Hz in the same ros node. The talker2 callback can be blocked if the talker1 callback takes more than 1s. Because `ros::spin()`  is single-threaded.
* Note: this issue only happens with roscpp. The rospy spinner is already multi-threaded, so you won’t be able to reproduce this issue if you translate the cpp subscriber node to Python.

### Solution One: MultiThreadedSpinner

* Blocking method. Simply just increase the number of threads to process callback queues.

```cpp
ros::MultiThreadedSpinner spinner(4); // Use 4 threads; or 0 to use all threads
spinner.spin(); // spin() will not return until the node has been shutdown
```

### Solution Two: AsyncSpinner

* Non-blocking method. Try to get each callback a thread if available.

```cpp
int main(int argc, char **argv) {
    ros::init(argc, argv, "talker_subscribers");
    ros::NodeHandle nh;
    ros::AsyncSpinner spinner(0); // 0 means max number of threads available
    spinner.start();  // start the AsyncSpinner asynchronously (non-blocking)
    ros::Subscriber counter1_sub = nh.subscribe("talker1", 10, callbackTalker1);
    ros::Subscriber counter2_sub = nh.subscribe("talker2", 10, callbackTalker2);
    ros::waitForShutdown(); // block the program; wait for the node to be killed
    // alternatively, you may have an infinite loop here to process some data
}
```

References: <http://wiki.ros.org/roscpp/Overview/Callbacks%20and%20Spinning>; <https://roboticsbackend.com/ros-asyncspinner-example>; <https://levelup.gitconnected.com/ros-spinning-threading-queuing-aac9c0a793f>


# ROS2


# Convert ROS1 bag to ROS2 bag

For rosbag files recorded in ROS1 environment, we cannot use them directly in ROS2 environment. There are some discussions on workarounds. In summary, there are two ways, as summarized in [this article](https://docs.ros.org/en/noetic/api/ov_core/html/dev-ros1-to-ros2.html).

## Solution 1: Install `rosbag_v2` plugin and `ros1_bridge`

* Requirement: Ubuntu 20 or below, where we have both ROS1 and ROS2 available.
* Follow the instructions in [ros1\_bridge](https://github.com/ros2/ros1_bridge) and [rosbag\_v2](https://github.com/ros2/rosbag2_bag_v2) to install.

```
sudo apt-get install ros-$ROS2_DISTRO-ros2bag ros-$ROS2_DISTRO-rosbag2*
sudo apt install ros-$ROS2_DISTRO-rosbag2-bag-v2-plugins
```

* We can then playback ros1 bag files in a ros2 environment.

```
source_ros1
source_ros2
ros2 bag play -s rosbag_v2 V1_01_easy.bag
```

Additional comments: For Ubuntu 22 or above environment, according to [ros2 official docs](https://docs.ros.org/en/humble/How-To-Guides/Using-ros1_bridge-Jammy-upstream.html), it may be possible to install ros1\_bridge alongside ros2, if ros2 is installed from source. In practice, I did not run through this successfully and got stuck at the `ros-core-dev : Depends: catkin but it is not installable` error.

References:

* <https://robotics.stackexchange.com/questions/101945/playback-ros1-bag-files-in-ros2>
* <https://answers.ros.org/question/402302/playback-ros1-bag-files-in-ros2/>
* <https://docs.ros.org/en/humble/How-To-Guides/Using-ros1_bridge-Jammy-upstream.html>

## Solution 2: Use `rosbags` to convert bag files (Recommended)

* This is a python package that does not depend on ros environment. We can convert a ros1 bag file into ros2 bag by running the following commands.

```
pip3 install rosbags>=0.9.11
rosbags-convert --src V1_01_easy.bag --dst <ros2_bag_folder>
```

References:

* <https://gitlab.com/ternaris/rosbags>
* <https://docs.ros.org/en/noetic/api/ov_core/html/dev-ros1-to-ros2.html>


# C++

* 30分钟了解C++11新特性 ([blog](https://blog.csdn.net/u013321328/article/details/24805691), [pdf](https://pan.baidu.com/s/1pHCNV))
* Modern C++ 11/14/17/20 Tutorial ([Chinese](https://changkun.de/modern-cpp/zh-cn/00-preface/), [English](https://changkun.de/modern-cpp/en-us/00-preface/), [Github](https://github.com/changkun/modern-cpp-tutorial), [website](https://changkun.de/modern-cpp/))


# C++ 11

## C++ 11

### auto

```cpp
auto a; // wrong
auto i = 1; 
auto d = 1.0; 
auto str = "Hello World"; 
auto ch = 'A'; 
auto func = less<int>(); 
vector<int> iv; 
auto ite = iv.begin();
auto p = new foo() // customized type
```

### initialization

```cpp
int arr[3]{1, 2, 3}; 
vector<int> v{1, 2, 3}; 
map<int, string> m{{1, "a"}, {2, "b"}};
string str{"Hello World"};
```

### for loop

```cpp
map<string, int> m{{"a", 1}, {"b", 2}, {"c", 3}}; 
for (auto p : m){ 
    cout<<p.first<<" : "<<p.second<<endl;
}
```

### tuple (variable-length template)

```cpp
auto p = make_pair(1, "before C++ 11");

auto t1 = make_tuple(1, 2.0, "C++ 11"); 
auto t2 = make_tuple(1, 2.0, "C++ 11", {1, 0, 2});
```

### lambda expression

```cpp
vector<int> iv{5, 4, 3, 2, 1}; 
int a = 2, b = 1;
// syntax: [global param](func args)->return_type{func body} 
for_each(iv.begin(), iv.end(), [b](int &x){cout<<(x + b)<<endl;});
for_each(iv.begin(), iv.end(), [=](int &x){x *= (a + b);});
for_each(iv.begin(), iv.end(), [=](int &x)->int{return x * (a + b);});
```

### decltype

```cpp
template <typename Creator> 
auto processProduct(const Creator& creator) -> 
decltype(creator.makeObject()) { 
    auto val = creator.makeObject(); 
    // do something with val
}
```

### nullptr

```cpp
void F(int a){
    cout << a << endl;
} 
void F(int *p){
    assert(p != NULL); 
    cout << p << endl;
} 
int main(){
    int *p = nullptr; 
    int *q = NULL; 
    int a = nullptr;  // compilation error; not allowed
    bool equal = ( p == q );  // true
    F(0);  // compilation error in C++98 due to ambiguity; call F(int) in C++11
    F(nullptr);
    return 0;
}
```

### rvalue reference; move semantics; perfect forwarding

```cpp
#include <iostream> 
#include <utility> 
void reference(int& v) { 
    std::cout << "lvalue" << std::endl;
} 
void reference(int&& v) { 
    std::cout << "rvalue" << std::endl;
} 
template <typename T> 
void pass(T&& v) { 
    std::cout << "regular: "; 
    reference(v); 
    std::cout << "std::move: "; 
    reference(std::move(v)); 
    std::cout << "std::forward: "; 
    reference(std::forward<T>(v)); // cf. static_cast<T&&>(v)
} 
int main() { 
    std::cout << "pass rvalue:" << std::endl; 
    pass(1);
    
    std::cout << "pass lvalue:" << std::endl; 
    int v = 1; 
    pass(v);
    
    return 0;
}
/* Output:
pass rvalue:
regular: lvalue
std::move: rvalue
std::forward: rvalue
pass lvalue:
regular: lvalue
std::move: rvalue
std::forward: lvalue
*/

```


# C++ Examples

## C++ Examples

### std::vector

* `std::vector<int> iv` by itself is on the stack, but its data is on the heap
* `sizeof(std::vector<int>)` is 24; it contains three 8-bytes pointers (on 64-bit machines)
  * one pointing to the beginning of the data on the heap
  * one pointing to the end of the data (size)
  * one pointing to the end of the allocated memory (capacity)
* starting from empty, the capacity can grow to `1 2 4 8 16 ...` under gcc compiler; it may differ under other compilers (e.g., MSVC)&#x20;
* essentially is a dynamic array: O(1) to insert at the end, but O(n) to insert elsewhere

```cpp
std::vector<int> v; 
for(std::size_t i=0; i<10; ++i){
    v.push_back(i);
    std::cout << v.size() << " " << v.capacity() << std::endl;
}

for(auto i=v.begin(); i!=v.end(); ++i){
    std::cout << &*i << std::endl;  // print the address of each element
}
```

references: <https://stackoverflow.com/questions/8036474/when-vectors-are-allocated-do-they-use-memory-on-the-heap-or-the-stack>;  <https://stackoverflow.com/questions/12271017/initial-capacity-of-vector-in-c>;  <https://chryswoods.com/beginning_c++/lists.html>;&#x20;

### static\_cast and dynamic\_cast

The static cast performs conversions between **compatible types**. It is similar to the C-style cast, but is more restrictive. For example, the C-style cast would allow an integer pointer to point to a char.

```cpp
char c = 10;       // 1 byte
int *p = (int*)&c; // 4 bytes
```

Since this results in a 4-byte pointer pointing to 1 byte of allocated memory, writing to this pointer will either cause a run-time error or will overwrite some adjacent memory.

```cpp
*p = 5; // run-time error: stack corruption
```

In contrast to the C-style cast, the static cast will allow the compiler to check that the pointer and pointee data types are compatible, which allows the programmer to catch this incorrect pointer assignment during compilation.

```cpp
int *q = static_cast<int*>(&c); // compile-time error
```

reference: <https://stackoverflow.com/questions/28002/regular-cast-vs-static-cast-vs-dynamic-cast/18414172#18414172>;

### template and typename

* `typename` needed by dependent names

```cpp
// example in a templated class
typedef pcl::PointCloud<PointInT> PointCloudIn;
typedef typename PointCloudIn::Ptr PointCloudInPtr;  // need typename due to Ptr
typedef typename PointCloudIn::ConstPtr PointCloudInConstPtr; // need typename

// example in a templated function
template <typename PointT> // needed for templated argument type dependent on PointT
void func(typename pcl::PointCloud<PointT>::Ptr& cloud){
  // if creating a new instance dependent on PointT
  typename pcl::KdTreeFLANN<PointT>::Ptr tree;
}

// example error message without typename keyword
//// error: need ‘typename’ before ‘pcl::KdTreeFLANN<PointT>::Ptr’ 
//// because ‘pcl::KdTreeFLANN<PointT>’ is a dependent scope

// one more example
template <typename FeatureT>
class MyPerception
{
  public:
    using PointT = pcl::PointXYZRGBNormal;
    using NormalT = pcl::PointXYZRGBNormal;

    using PointCloudPtr = pcl::PointCloud<PointT>::Ptr;         // not needed
    using KeypointPtr = pcl::PointCloud<PointT>::Ptr;           // not needed
    using FeaturePtr = typename pcl::PointCloud<FeatureT>::Ptr; // needed
};
```

reference: <https://stackoverflow.com/questions/610245/where-and-why-do-i-have-to-put-the-template-and-typename-keywords/17579889#17579889>

* `template` keyword is needed to tell the compiler explicitly that `.cast` is a template

```cpp
// In PCL gicp.hpp
// I would like to save the transformation matrix in each iteration for visualization
rigid_transformation_estimation_(output, source_indices, *target_, target_indices, transformation_);
transformations_.push_back(transformation_.template cast<double>());
```

references: [Stackoverflow: When casting Eigen matrix type, error: expected primary-expression before ‘float’](https://stackoverflow.com/a/48029026/13980439)

### const

```cpp
class MyClass {
  public:
    MyClass (): point(0.1) {};
    typedef boost::shared_ptr<MyClass> Ptr;
    typedef boost::shared_ptr<const MyClass> ConstPtr;
    void mem_func (MyClass::Ptr ptr) const;
    float point;
};

// not allowed to change the member variables of MyClass::Ptr (which is the raw pointer)
void func(const MyClass::Ptr ptr){
  ptr->point = 1.0; // OK
  ptr.reset();      // error: not able to reset to null or to other instance
}

// not allowed to change the member variables of MyClass
void func(MyClass::ConstPtr ptr){
  ptr->point = 1.0; // error: not able to change the value
  ptr.reset();      // OK
}

void MyClass::mem_func(MyClass::Ptr ptr) const {
  point = 1.0;        // error: not allowed to change member variables
  this->point = 1.0;  // error: not allowed to change member variables
  ptr->point = 1.0;   // OK  (external pointer)
}
```

### mutable

Recall that the `const` reference or pointer is constrained to

* only read access for any visible data members,
* permission to call only methods that are marked as `const` (because non-const member functions can write to data members, which breaks the fence).

The `mutable` keyword can allow you to modify a variable in a `const` method. It is like making exceptions for a few special variables.

```cpp
class Foo {
  private:
    mutable bool done_;
  public:
    void doSomething() const { ...; done_ = true; }
};
```

references: <https://stackoverflow.com/questions/105014/does-the-mutable-keyword-have-any-purpose-other-than-allowing-the-variable-to>;&#x20;

### reference

```cpp
void func (MyClass::Ptr ptr){
  std::cout << "use_count() = " << ptr.use_count() << std::endl; // 2
}

void func_ref (MyClass::Ptr& ptr){
  std::cout << "use_count() = " << ptr.use_count() << std::endl; // 1
}

int main(){
  MyClass::Ptr pointer (new MyClass);
  std::cout << "use_count() = " << pointer.use_count() << std::endl; // 1
  func (pointer);
  func_ref (pointer);
}
```

### virtual function (for runtime polymorphism)

```cpp
// flann/algorithms/nn_index.h
virtual int knnSearch(const Matrix<ElementType>& queries,
                            Matrix<size_t>& indices,
                            Matrix<DistanceType>& dists,
                            size_t knn,
                      const SearchParams& params) const
{
  // implementation
}

int knnSearch(const Matrix<ElementType>& queries,
                    Matrix<int>& indices,
                    Matrix<DistanceType>& dists,
                    size_t knn,
              const SearchParams& params) const
{
	flann::Matrix<size_t> indices_(new size_t[indices.rows*indices.cols], indices.rows, indices.cols);
	int result = knnSearch(queries, indices_, dists, knn, params);
  // wrapper
}
```

### templated classes in cpp/hpp files & explicit instantiation

* **behavior:** the templated classes can only be implemented in hpp files, but not cpp files
* **workaround:** add explicit instantiation in the cpp files
* **reason in short:** a template is literally a template; a class template is not a class, it's a recipe for creating a new class for each T we encounter

references: [splitting templated classes into cpp/hpp files](https://stackoverflow.com/questions/1724036/splitting-templated-c-classes-into-hpp-cpp-files-is-it-possible/9992920); [why can templated only be implemented in the header file](< https://stackoverflow.com/questions/495021/why-can-templates-only-be-implemented-in-the-header-file>); [explicit template instantiation when is used](https://stackoverflow.com/questions/2351148/explicit-template-instantiation-when-is-it-used)

### smart pointer vs. raw pointer

```cpp
pcl::PointCloud<PointT> cloud;  // instance
pcl::PointCloud<PointT>::Ptr cloud_ptr (new pcl::PointCloud<PointT>); //smart pointer
auto cloud_raw_ptr = new pcl::PointCloud<PointT>;  // raw pointer 

*cloud_ptr      // instance
*cloud_raw_ptr  // instance   typeid(*cloud_ptr) == typeid(*cloud_raw_ptr)
&cloud          // raw pointer
&(*cloud_ptr)   // will turn the smart pointer into a raw pointer

std::cout << typeid(*cloud_ptr).name() << std::endl;
```

### iterator

There are five types of iterator available in STL.

![](/files/-Mgw6VEj1hckjra-7BEe)

* Bidirectional Iterator available for `std::list`,`std::set` and `std::map`
* Random Access Iterator available for `std::vector` and `std::deque`
* No iterator available for `std::stack`, `std::queue` and `std::priority_queue`&#x20;

```cpp
std::vector<int> vec{1, 2, 3, 4, 5};
std::vector<int>::iterator it;
for (it = vec.begin(); it < vec.end(); ++it)
    std::cout << *it << "\n";
// begin() points to the first element in the container
// end() points to the element after the last element (empty)
```

### function pointer

```cpp
// in keypoint.h
typedef boost::function<int (int, double, std::vector<int> &, std::vector<float> &)> SearchMethod;

// class Search in search.h
virtual int
radiusSearch (int index, double radius, std::vector<int> &k_indices,
              std::vector<float> &k_sqr_distances, unsigned int max_nn = 0) const;

// in pcl::Keypoint<PointInT, PointOutT>::initCompute ()
int (KdTree::*radiusSearch)(int index, double radius, std::vector<int> &k_indices,
        std::vector<float> &k_distances, unsigned int max_nn) const = &KdTree::radiusSearch;
search_method_ = boost::bind (radiusSearch, boost::ref (tree_), _1, _2, _3, _4, 0);
// boost::ref is useful for passing references to function templates (algorithms)
// that would usually take copies (values) of their arguments.

// the simplest function pointer
void (*fun_ptr)(int) = &fun; // fun_ptr is a pointer to function fun() 
```

reference: [Stackoverflow: how to use boost bind with a member function](https://stackoverflow.com/a/68188191/13980439); [GeeksForGeeks: function pointer in C](https://www.geeksforgeeks.org/function-pointer-in-c/); [Function Pointers in C and C++](https://www.cprogramming.com/tutorial/function-pointers.html)

### ostream

* keep in mind: streams are not copyable, but are movable
  * so that the copy constructor of any object that owns a stream won't work
  * because if a class has a non-copyable member then the containing class is also non-copyable

```cpp
#include <iostream>
#include <fstream>

void myprogram(std::ostream& os) {   // not const
    os << "Hi" << std::endl;
}

int main() {
    myprogram(std::cout);
    std::ofstream ofs("example.txt");  // constructor opens the file
    if(ofs.is_open())
        myprogram(ofs);
    return 0;                          // destructor closes the file
}
```

reference: [Sending cout to a log file](http://www.cplusplus.com/forum/beginner/48427/)

### template specialization

* Explicit (full) template specialization: Allows customizing the template code for a given set of template arguments.
* Partial template specialization: Allows customizing class \[and variable (since C++14)] templates for a given category of template arguments.

```cpp
// 1) Template declaration (no specialization)
template <class Key, class Value>
struct Foo {};

// 2) Partial specialization
template <class Key>
struct Foo<Key, int> {};

// 3) Full/explicit specialization
template <>
struct Foo<std::string, int> {};

// Then, when instantiating the templates, 
// the compiler will choose the most specialized definition available:
Foo<std::string, std::string> f1; // Should match #1
Foo<int,         int>         f2; // Should match #2
Foo<std::string, int>         f3; // Should match #3
// #1 and #3 work for template functions as well.

// Examples in OpenCV (traits.hpp)
template<typename _Tp> class DataType
{
public:
    typedef _Tp         value_type;
    ...
};
template<> class DataType<bool> { ... };
template<> class DataType<uchar>  { ... };
template<> class DataType<char>  { ... };
...
```

references: [Stackoverflow: Template specialization with empty brackets and struct](#static_cast-and-dynamic_cast); [cppreference: Explicit (full) template specialization](#c++-examples); [cppreference: Partial template specialization](https://en.cppreference.com/w/cpp/language/partial_specialization); [include/opencv2/core/traits.hpp](https://github.com/opencv/opencv/blob/master/modules/core/include/opencv2/core/traits.hpp)

### enum vs union

* Enum: helps to assign constants to a set of names to make program easier to read, maintain and understand
  * Enums are not actual variables; they're just a semi-type-safe form of #define. They're a way of storing a number in a reader-friendly format. The compiler will transform all uses of an enumerator into the actual numerical value.
* Union: helps to store data of different types as a single unit

```cpp
// Enum example in OpenCV (traits.hpp)
// template<> class DataType<bool>
enum { generic_type = 0,
       depth        = CV_8U,
       channels     = 1,
       fmt          = (int)'u',
       type         = CV_MAKETYPE(depth, channels)
     };

// Enum example in point cloud registration
enum class RegistrationMethod { PCL, MTPCL, FAST };
struct RegistrationResult {
  RegistrationMethod method;
  double time_elapsed;
  double fitness;
  Eigen::Matrix4d transformation;
};

// Union example in PCL (point_types.hpp)
// PointXYZLAB
union
{
  struct
  {
    float L;
    float a;
    float b;
  };
  float data_lab[4];
};
```

references: [Stackoverflow: what is the size of an enum type data in C++?](https://stackoverflow.com/a/8115895/13980439);&#x20;

### return multiple values

For returning two values we can use a `std::pair` (usually typedef'd). In C++11 and newer, there's `std::tuple` for more than two return results. With introduction of structured binding in C++17, returning `std::tuple` should probably become accepted standard.

In practice, if multiple result values are needed, the best practice is to create a struct to store these values and return this struct or change it as the pass-by-reference argument.

<pre class="language-cpp"><code class="lang-cpp">// in C++ 11
#include &#x3C;tuple>
std::tuple&#x3C;int, int> divide(int dividend, int divisor) {
    return std::make_tuple(dividend / divisor, dividend % divisor);
}
#include &#x3C;iostream>
int main() {
    int quotient, remainder;
    std::tie(quotient, remainder) = divide(14, 3);
}
// Example in Open3D
// open3d/pipelines/registration/ColoredICP.cpp
Eigen::Matrix6d JTJ;
Eigen::Vector6d JTr;
double r2;
std::tie(JTJ, JTr, r2) =
        utility::ComputeJTJandJTr&#x3C;Eigen::Matrix6d, Eigen::Vector6d>(
                compute_jacobian_and_residual, (int)corres.size());
// open3d/utility/Eigen.cpp
template &#x3C;typename MatType, typename VecType>
std::tuple&#x3C;MatType, VecType, double> ComputeJTJandJTr(
        std::function&#x3C;void(int, VecType &#x26;, double &#x26;, double &#x26;)> f,
        int iteration_num,
        bool verbose /*=true*/) { ... }
<strong>// Example in TEASER++
</strong>namespace teaser {
struct RegistrationSolution {
  bool valid = true;
  double scale;
  Eigen::Vector3d translation;
  Eigen::Matrix3d rotation;
  EIGEN_MAKE_ALIGNED_OPERATOR_NEW
};
}
</code></pre>

references: [StackOverflow: Returning multiple values from a C++ function](https://stackoverflow.com/questions/321068/returning-multiple-values-from-a-c-function); [Open3D/ColoredICP](https://github.com/isl-org/Open3D/blob/master/cpp/open3d/pipelines/registration/ColoredICP.cpp); [TEASER-plusplus/include/teaser/registration.h](https://github.com/MIT-SPARK/TEASER-plusplus/blob/master/teaser/include/teaser/registration.h#L30)

### std::thread

* template< class Function, class... Args > explicit thread( Function&& f, Args&&... args );
* Move constructor is allowed, but copy constructor is deleted.
* If running class member functions, an instance of the class is needed as the first argument.

```cpp
// Example from slambook2 ch13
Viewer::Viewer() {
    viewer_thread_ = std::thread(std::bind(&Viewer::ThreadLoop, this));
}
void Viewer::Close() {
    viewer_running_ = false;
    viewer_thread_.join();
}

// Example from LeGO-LOAM mapOptimization
int main(int argc, char** argv) {
    ros::init(argc, argv, "lego_loam");
    mapOptimization MO;
    std::thread loopthread(&mapOptimization::loopClosureThread, &MO);
    std::thread visualizeMapThread(&mapOptimization::visualizeGlobalMapThread, &MO);
    ros::Rate rate(200);
    while (ros::ok()) {
        ros::spinOnce();
        MO.run();
        rate.sleep();
    }
    loopthread.join();
    visualizeMapThread.join();
    return 0;
}

// Example from A-LOAM laserMapping
void process() {
    while(1) { ... }
}
int main(int argc, char **argv) {
    ros::init(argc, argv, "laserMapping");
    std::thread mapping_process{process};
    ros::spin();
    return 0;
}
```

references: [C++ reference on std::thread constructor](https://en.cppreference.com/w/cpp/thread/thread/thread); [slambook2 chapter 13 source code](https://github.com/hanzheteng/slambook2/blob/master/ch13/src/viewer.cpp#L14); [LeGO-LOAM map optimization source code](https://github.com/hanzheteng/LeGO-LOAM/blob/master/LeGO-LOAM/src/mapOptmization.cpp#L1534); [A-LOAM mapping source code](https://github.com/hanzheteng/A-LOAM/blob/devel/src/laserMapping.cpp#L957);&#x20;

### volatile

* It is a keyword to tell compiler that this variable may be modified from outside the program which compiler is not aware of. Therefore, please do not optimize this variable.
* It is a qualifier that can also be applied to methods/functions.
* `volatile` and `const` are two faces of the same coin.

```cpp
// Example without volatile
int some_int = 100;
while (some_int == 100) { // compilier may optimize this to while(true)
   // code not related to some_int
}

// Example with volatile
volatile int some_int = 100; // tells compiler not to optimize this variable
while (some_int == 100) {
   // code not related to some_int
}
```

references: [C++ keywords: volatile](https://en.cppreference.com/w/cpp/keyword/volatile); <https://stackoverflow.com/questions/4437527/why-do-we-use-volatile-keyword>; <https://stackoverflow.com/questions/4479597/does-making-a-struct-volatile-make-all-its-members-volatile/4479652>;&#x20;

### memory fence/barrier

* This refers to a set of instructions to synchronize memory access (read/writes occur in the order you expect). For example a 'full fence' means all read/writes before the fence are comitted before those after the fence. It is a hardware level concept, and in higher level languages we are used to dealing with `mutex`es and `semaphore`s.&#x20;
* Note that the keyword `volatile` *does not guarantee a memory barrier* to enforce cache-consistency. Although It is guaranteed that volatile reads/writes will happen in the exact order specified in the source code, but volatile reads/writes can still be reordered with respect to non-volatile ones. (This is referred to as compiler reordering optimizations.)

Thread #1 Core #1:

```cpp
 while (f == 0);
 // Memory fence required here
 print x;
```

Thread #2 Core #2:

```cpp
 x = 42;
 // Memory fence required here
 f = 1;
```

references: <https://stackoverflow.com/questions/1525189/do-i-need-a-mutex-for-reading>; <https://en.wikipedia.org/wiki/Memory_barrier>; <https://stackoverflow.com/questions/286629/what-is-a-memory-fence>;&#x20;

### hash

By default, `std::pair` and `std::tuple` are not hashable, and therefore they cannot serve as keys in `std::unordered_map`. For hashing and combining multiple values, one way is to use `boost::hash_combine` function. Alternatively, we can write our own hash functions.

```cpp
/**
 * @brief Templated hash function for Eigen::Matrix class.
 * 
 * Example usage:
 *   std::unordered_map<Eigen::Vector3i, double, hash_matrix<Eigen::Vector3i>>
 * 
 * @note This implementation is oblivious to the storage order of Eigen matrix 
 * (column- or row-major). It will give you the same hash value for two different
 * matrices if they are the transpose of each other in different storage order.
 * 
 *  The code is from `hash_combine` function of the Boost library.
 */
template<typename T>
struct hash_matrix : std::unary_function<T, std::size_t> {
  std::size_t operator()(T const& matrix) const {
    size_t seed = 0;
    for (size_t i = 0; i < matrix.size(); ++i) {
      auto elem = *(matrix.data() + i);
      seed ^= std::hash<typename T::Scalar>()(elem) + 0x9e3779b9 + (seed << 6) + (seed >> 2);
    }
    return seed;
  }
};

/**
 * @brief An alternative hash function optimized to reduce collision rate for 
 * numerous vectors that all contain elements from a small continuous distribution.
 */
std::size_t operator()(std::vector<uint32_t> const& vec) const {
  std::size_t seed = vec.size();
  for(auto x : vec) {
    x = ((x >> 16) ^ x) * 0x45d9f3b;
    x = ((x >> 16) ^ x) * 0x45d9f3b;
    x = (x >> 16) ^ x;
    seed ^= x + 0x9e3779b9 + (seed << 6) + (seed >> 2);
  }
  return seed;
}
```

References: <https://stackoverflow.com/questions/20511347/a-good-hash-function-for-a-vector/>; <https://stackoverflow.com/questions/17016175/c-unordered-map-using-a-custom-class-type-as-the-key>; <https://wjngkoh.wordpress.com/2015/03/04/c-hash-function-for-eigen-matrix-and-vector/>

### class inheritance and polymorphism

```cpp
pcl::Registration<PointT, PointT>::Ptr base_ptr;
if (method == "pcl")
  base_ptr = std::make_shared<pcl::GeneralizedICP<PointT, PointT>>();
else if (method == "pclmt")
  base_ptr = std::make_shared<pcl::MultithreadedGeneralizedICP<PointT, PointT>>();
else if (method == "fast")
  base_ptr = std::make_shared<fast_gicp::FastGICP<PointT, PointT>>();
else
  throw std::runtime_error("unknown gicp method");
// Then use base_ptr to call functions in derived class
// Note: these functions must already exist in base class as virtual functions
base_ptr->setInputSource(source);
base_ptr->setInputTarget(target);
```

### factory method

The most basic factory method&#x20;

```cpp
unique_ptr<Animal> makeAnimal(const string& type, int number) {
  if (type == "Dog") return make_unique<Dog>(number);
  if (type == "Cat") return make_unique<Cat>(number);
  throw runtime_error("Invalid type!")
}
```

Reference: [Unforgettable Factory Registration](http://www.nirfriedman.com/2018/04/29/unforgettable-factory/)


# C++ Debug

## Debug FAQ

### error: if condition proceeds with a bool variable that is false

```cpp
fake_indices_ = false; // member variable initialized somewhere else

bool MultiviewGICP::initCompute() {
  std::cout << "fake_indices_ = " << fake_indices_ << std::endl;
  // this prints 0, and confirms that fake_indices_ was indeed initialized as false
  if (fake_indices_) {
    std::cout << "we are getting here." << std::endl;
    // the weird thing is that this line of code gets printed out,
    // meaning that we reached this code block, suppressing the if condition
  }
  // the root cause of this problem was the missing of the return statement below,
  // which is required of this function.
  // Without it, program proceeds with whatever code being compiled in the memory.
  return true;
  // if you put directly "if (false)", the above code block will not be executed,
  // since they may be optimized out during compilation.
}
```

### error: variable or field declared void

```cpp
// for example it happens in this case
void initializeJSP(unknownType Experiment);

// another example (due to the missing 'typename' keyword)
template <typename PointT>
void func(pcl::PointCloud<PointT>::Ptr& cloud) {} // error
void func(typename pcl::PointCloud<PointT>::Ptr& cloud) {} // correct
```

reference: <https://stackoverflow.com/questions/364209/variable-or-field-declared-void>


# Factory Method

## Unforgettable Factory Registration

by Nir's blog 29 Apr 2018 ([original link](http://www.nirfriedman.com/2018/04/29/unforgettable-factory/))

Using a factory is a common pattern when we are working with polymorphic objects. It exists to solve a very basic issue in C++: in order to construct something you must name its type. But the entire point of runtime polymorphism is often that we cannot name the type, because we won’t know it until runtime.

The most basic factory would just consist of some `if` statements coupled together:

```cpp
unique_ptr<Animal> makeAnimal(const string& type, int number)
{
  if (type == "Dog") return make_unique<Dog>(number);
  if (type == "Cat") return make_unique<Cat>(number);
  throw runtime_error("Invalid type!")
}
```

This often isn’t a great solution. In particular, we keep having to update this central piece of code every time a new derived class gets written. Under the best of circumstances, this is merely irritating and code smell. However, in some cases it can be much worse. It’s common for a library to provide an interface and a few concrete classes deriving from it. If the library uses a factory somewhere that’s written like this, it will be impossible for the user to inject their own classes into the library.

Aside from very simple use cases, better factories tend to predicated on allowing classes to register themselves for construction by the factory; this allows library code to construct user classes that were written afterwards, and avoids the issue of a central function that needs to change with each new class. This is usually by done changing the logic of the factory from code—multiple `if`s—into data, in particular an associative map. It usually looks something like this:

```cpp
unordered_map<string, unique_ptr<Animal>(*)(int)> factory_map;
factory_map["Dog"] = Dog::make;
factory_map["Cat"] = Cat::make;
unique_ptr<Animal> makeAnimal(const string& type, int number)
{
  return factory_map.at(type)(number);
}
```

### Registration <a href="#registration" id="registration"></a>

Using an associative map for this sort of thing is the way to go, but there’s still the whole issue of registration. It’s easier to forget to do, or do incorrectly. Overwhelmingly, I’ve seen macros used for this purpose; something along the lines of:

```cpp
class Dog : public Animal {
};
REGISTER_CLASS(factory_map, Dog);
```

It has to be done manually for each derived class and it’s easy to forget. We’re going to look at how to automate this process, removing boilerplate and eliminating the possibility of mistakes.

### Goal <a href="#goal" id="goal"></a>

Wouldn’t it be magical, if instead of worrying about macros, or some global dictionary, we instead could just push everything into a library? Users could just write code similar to this:

```cpp
struct Animal : Factory<Animal, int> {
  virtual void makeNoise() = 0;
  ...
};

class Dog : public Animal::Registrar<Dog> {
public:
  Dog(int x);
  void makeNoise() override;
  ...
};

auto x = Animal::make("Dog", 3);
x->makeNoise();
```

No macros, no magic, no problems. The end user is just solving their problem. Does this seem to good to be true? Well, it’s not.

### Solution Sketch <a href="#solution-sketch" id="solution-sketch"></a>

Let’s start by a sketch of some of the functionality that we want. Because this is all going to be automated, I’m going to opt to simply inject the factory interface directly into the base class using the Curiously Recurring Template Pattern ([CRTP](https://eli.thegreenplace.net/2011/05/17/the-curiously-recurring-template-pattern-in-c)), instead of having it as a separate entity.

```cpp
template <class Base, class... Args>
class Factory {
public:
  template <class ... T>
  static unique_ptr<Base> make(const string &s, T&&... args) {
      return data().at(s)(forward<T>(args)...);
  }

  friend Base;

private:
  using FuncType = unique_ptr<Base> (*)(Args...);
  Factory() = default;

  static auto &data() {
    static unordered_map<string, FuncType> s;
    return s;
  }
};
```

The factory is templated on the base class that it is both injecting interface into, and that it produces unique pointers to. The `Args` template parameter represent the arguments required to produce a derived instance from the factory. Note that we use the common CRTP trick of making the constructor `private`, and the template a `friend` to make it more difficult to misuse (`class Foo : Factory<Bar, int>` will not compile).

So far, so good. But we haven’t dealt with registration at all. So let’s take a stab at it. The two techniques to use are:

1. CRTP to inject functionality into derived classes automatically.
2. Use initialization of a static member to force code to be executed before main.

Let’s give this a shot then; we’ll declare a nested class inside `Factory`:

```cpp
template <class Base, class... Args>
class Factory {
...
  template <class T>
  struct Registrar : Base {
    friend T;

    static bool registerT() {
      const auto name = T::name;
      Factory::data()[name] =
          [](Args... args) -> unique_ptr<Base> {
        return make_unique<T>(forward<Args>(args)...);
      };
      return true;
    }
    static bool registered;

  private:
    Registrar() = default;
  };
};

// The really fun part
template <class Base, class... Args>
template <class T>
bool Factory<Base, Args...>::Registrar<T>::registered =
    Factory<Base, Args...>::Registrar<T>::registerT();
```

The intention here is that classes wishing to implement the `Base` interface will inherit from `Registrar` instead of from `Base` directly. This will instantiate the template, including the `registered` member, causing it to get initialized and causing the derived class to get registered. We assume (for now) that the derived class provides a static member `name` to indicate how it would like to be named in the factory.

One quick note from the above is the use of `forward`. This is not actual perfect forwarding because `Args` are fixed by the class and not deduced by the function. However it is still necessary so that value and rvalue reference types get forwarded correctly.

There’s just one little problem: 1) and 2) are not compatible with one another! The CRTP pattern involves templates, and unused members of class templates are not instantiated. This seems annoying but it’s quite useful in other contexts: it allows us to write class templates that may have only part of their members usable for certain parameters; for example a `vector` will not be copyable if its contained type is not copyable, which works because the copy constructor is not instantiated unless it’s used.

So, we need to make sure that `registered` gets used. But of course, we have to use it from some part of the class that is itself guaranteed to be used. What’s guaranteed to be used? Well, `Registrar`’s constructor will have to be instantiated, so let’s use that:

```cpp
  ...
  private:
    Registrar() { (void)registered; }
```

Pretty strange looking, but this code will work. Before we get a feel for using it, we’re going to make some improvements.

### Automatic name <a href="#automatic-name" id="automatic-name"></a>

One nice change would be to obviate the necessity of the derived member having a static `name` member. As it turns out, we can do this, at least if you’re not disabling RTTI. Based on the discussion [here](https://stackoverflow.com/questions/281818/unmangling-the-result-of-stdtype-infoname), we can write:

```cpp
string demangle(const char *name) {

  int status = -4;

  unique_ptr<char, void (*)(void *)> res{
      abi::__cxa_demangle(name, NULL, NULL, &status), free};

  return (status == 0) ? res.get() : name;
}
```

We can then change `registerT`

```cpp
    static bool registerT() {
      const auto name = demangle(typeid(T).name());
      Factory::data()[name] = [](Args... args) -> unique_ptr<Base> {
        return make_unique<T>(forward<Args>(args)...);
      };
      return true;
    }
```

As I’ve implemented it here, the name will include the namespace. Obviously that’s simple enough to remove if you don’t want it to include that.

### Better safety <a href="#better-safety" id="better-safety"></a>

We’ve already employed the CRTP trick of making the constructor private and make the intended derived a friend, so that it’s impossible to accidentally template on something else when inheriting from the CRTP class. However, so far nothing is stopping users from inheriting directly from the base class, in which case no registration would occur. Preventing this with friendship is a bit tricky due to the various nested template classes, but we use a slightly different approach called the [Passkey](https://stackoverflow.com/questions/3217390/clean-c-granular-friend-equivalent-answer-attorney-client-idiom/3218920#3218920) idiom.

We declare another nested class inside `Factory`, and slightly change the constructor of Registrar:

```cpp
template <class T, class ... Args>
class Factory {
  ...
  template <class T>
  struct Registrar : Base {
  ...
  private:
    Registrar() : Base(Key{}) { (void)registered; }
  };
  ...
private:
  class Key {
    Key(){};
    template <class T> friend struct Registrar;
  };
};
```

The user base class will need to declare the constructor to take a `Key`, which in turn can only be created by `Registrar`. Thus, no class can derive without going through `Registrar`.

### Usage <a href="#usage" id="usage"></a>

Finally, after building up that whole structure, we can see what user code looks like. Remember that we have potentially two distinct users: the first user may be a library that wants to declare an interface, instantiates objects fulfilling the interface through a factory, and wants to allow clients to inject their own classes. The second user is a client of said library, who wants to create their own implementations of the interface. Of course, they could also be the same person as well; just a single developer trying to cleanly separate concerns.

Our final code looks pretty well identical to our dream that discussed in the Goal section! The first piece of user code is the base class. That code could look like this:

```cpp
struct Animal : Factory<Animal, int> {
  Animal(Key) {}
  virtual void makeNoise() = 0;
  virtual ~Animal() = default;
};
```

Every line of code here is actually expressing something relevant to their design, with the exception of line 2, which is a small price to pay to make sure that users of this base class don’t accidentally inherit directly, and then try to understand why their class isn’t registered.

Now let’s look at the next piece of user code, a derived class:

```cpp
class Dog : public Animal::Registrar<Dog> {
public:
  Dog(int x) : m_x(x) {}

  void makeNoise() override { cerr << "Dog: " << m_x << "\n"; }

private:
  int m_x;
};
```

Every single line of code here implements some kind of functionality for the derived class; boilerplate here is at an absolute minimum. Beyond automatic registration, we also protect at compile time against a number of errors:

* It’s not possible to have `Dog` inherit from `Animal` directly
* It’s not possible to have `Dog` inherit from `Registrar<Cat>`.
* It’s not possible to provide a constructor with an incompatible signature.

Finally, let’s take a look at a trivial bit of code that makes use of all this. Let’s assume that we have a `Cat` class as well defined by `:s/Dog/Cat` on our previous bit of code. Then we can write:

```cpp
int main() {
  auto x = Animal::make("Dog", 3);
  auto y = Animal::make("Cat", 2);
  x->makeNoise();
  y->makeNoise();
}
```

Which prints out:

```
Dog: 3
Cat: 2
```

as you would expect. See a full copy of the working code [here](http://coliru.stacked-crooked.com/a/11473a649e402831).

We’ve managed here to separate out much of the boilerplate that often goes with writing polymorphic code in C++, with a relatively small and simple amount of code. As we can see, the user code that defines the interface and various implementations can focus on the required logic, and get polymorphic construction for free.

Reference: [Unforgettable Factory Registration](http://www.nirfriedman.com/2018/04/29/unforgettable-factory/)


# Timing

## Timing

```cpp
// glog 0.09 ms
LOG(INFO) << "test";   // 0.09 ms

// steady_clock 0.000068 ms
static std::chrono::steady_clock::time_point tic = std::chrono::steady_clock::now();
std::chrono::steady_clock::time_point toc = std::chrono::steady_clock::now();
auto time_elapsed = std::chrono::duration_cast<std::chrono::nanoseconds>(toc - tic);
double time_ms = time_elapsed.count() / 1000000.0;
tic = toc;

// std::hash  0.000050 ms (avg from 10 times)
auto result = std::hash<std::string>{}("centroid start");

// map index  0.000100 ms
std::unordered_map<std::string, double> value;
value["test"];

// get filename 0.001 ms
std::cout << __FILE__ << "\n";

// get line number 0.004 ms
std::cout << __LINE__ << "\n";

// cout  0.0014 ms
std::cout << "[TIMING] " << note << ": " << time_ms << " ms" << std::endl;

// printf  0.0013 ms  // using printf with std::string is not recommended
printf("[TIMING] %s: %f ms\n", note.c_str(), time_ms);

// cin  0.025 ms  (with sync disabled 0.015 ms)
std::cin >> test;

// scanf 0.018 ms
scanf("%s", &test);
```

### **Tricks to run cin/cout faster**

* By default, `cin`/`cout` waste time synchronizing themselves with the C library’s stdio buffers, so that you can freely intermix calls to `scanf`/`printf` with operations on `cin`/`cout`. [Turn this off](http://gcc.gnu.org/onlinedocs/libstdc++/manual/io_and_c.html) with `std::ios_base::sync_with_stdio(false);`.
* Many C++ tutorials tell you to write `cout << endl` instead of `cout << '\n'`. But `endl` is actually slower because it forces a flush, which is usually unnecessary. (You’d need to flush if you were writing, say, an interactive progress bar, but not when writing a million lines of data.) Write `'\n'` instead of `endl`.
* There was a [bug](http://gcc.gnu.org/bugzilla/show_bug.cgi?id=7076) in very old versions of GCC (pre-2004) that significantly slowed down C++ iostreams. Don’t use ancient compilers.

Avoid these pitfalls, and `cin`/`cout` will be just as fast as `scanf`/`printf`. Some benchmarks indicate that `cin`/`cout` is **actually 5%–10% faster**. This is probably because `scanf`/`printf` need to interpret their format string argument at runtime.

However, I used to have a problem [Maximum Identity Matrix](https://codeforces.com/group/5ZOYcN1VgK/contest/324181/problem/C) on CodeForces that only `scanf` can pass but not `cin` (with sync disabled already; exactly the same code elsewhere).

```cpp
// pass case with scanf
#9: Accepted [140 ms, 4 MB, 1 points]
#10: Accepted [483 ms, 16 MB, 4 points]
#11: Accepted [15 ms, 0 MB, 1 points]
#12: Accepted [483 ms, 16 MB, 3 points]
#13: Accepted [514 ms, 16 MB, 3 points]

// failure case with cin
#9: Accepted [514 ms, 4 MB, 1 points]
#10: Time limit exceeded [1500 ms, 16 MB, 0 points]
#11: Accepted [15 ms, 0 MB, 1 points]
#12: Time limit exceeded [1500 ms, 16 MB, 0 points]
#13: Time limit exceeded [1500 ms, 16 MB, 0 points]
```

**References**

* [Stackoverflow: fastest way to get a timestamp](https://stackoverflow.com/questions/48609413/fastest-way-to-get-a-timestamp)
* [Quora: Is cin/cout slower than scanf/printf?](https://www.quora.com/Is-cin-cout-slower-than-scanf-printf)
* [Stackoverflow: Using scanf() in C++ programs is faster than using cin?](https://stackoverflow.com/questions/1042110/using-scanf-in-c-programs-is-faster-than-using-cin)

## Loop Timer

```cpp
#pragma once

#include <vector>
#include <chrono>
#include <iostream>
#include <iomanip>

namespace my {

/**
 * @brief LoopTimer is a timing tool optimized for high frequency loops (e.g., millions of iterations in a short time),
 * where any redundant computations or I/O operations can impact timing accuracy. Therefore, they should be avoided and
 * saved to be executed after the loop. This class is implemented based on chrono::steady_clock, which requires a
 * minimum execution time (e.g., 0.068 us) to place a time stamp. By placing multiple "stamps" in the loop/function,
 * LoopTimer can compute the average running time in between every two consecutive stamps, while minimizing additional
 * time consumption caused by the timing operation itself.
 * 
 * Example code:
 * 
 * @code
 * function_to_analyze() {
 *   my::LoopTimer timer(10);  // plan to place at most 10 stamps; this also starts the timer
 *   ...
 *   timer.stamp(0);
 *   for (int i = 0; i < 100000; ++i) {
 *     timer.stamp(1);  // place here to serve as a timing startpoint; in this case, 0-1 duration is not important
 *     ...
 *     timer.stamp(2);  // ends 1-2 duration and starts 2-3 duration
 *     ...
 *     timer.stamp(3);
 *     ...
 *   }
 *   timer.stamp(4);  // we end up using 5 stamps
 *   timer.printTimeMilliseconds();  // compute and print the average running time between every two stamps
 * }
 * @endcode
 * 
 * @author Hanzhe Teng
 * @date Aug 10, 2021
 */
class LoopTimer {
 public:
  /**
   * @brief Constructor. Initialize the number of stamps, such that memory can be allocated accordingly.
   * It will also start a new time duration automatically.
   * @param num_stamp the total number of stamps to be used
   */
  LoopTimer (int num_stamp)
      : num_stamp_ (num_stamp),
        counter_(num_stamp),
        sum_time_(num_stamp) {
    tic_ = std::chrono::steady_clock::now();
  }

  /**
   * @brief Place a time stamp to end the previous duration (toc) and start a new duration (tic).
   * @param idx the index of time stamp; stamps with the same index will be grossed and averaged.
   * @note No sanity check on index for the sake of performance.
   * This function requires minimal execution time, which is only 0.068 us for example.
   */
  inline void stamp(int idx = 0) {
    toc_ = std::chrono::steady_clock::now();
    counter_[idx] += 1;
    sum_time_[idx] += std::chrono::duration_cast<std::chrono::nanoseconds>(toc_ - tic_);
    tic_ = toc_;
  }

  /**
   * @brief Print the average running time in each duration and how many times each duration has been recorded.
   * @note The unit of time is second (s). This function should be called only once, after the loop.
   */
  void printTimeSeconds() const {
    for (int idx = 0; idx < num_stamp_; ++idx) {
      if (!counter_[idx]) continue;
      double time = sum_time_[idx].count() / 1000000000.0 / counter_[idx];
      std::cout << "[LoopTimer] stamp" << std::setw(3) << idx << ":  " << std::setw(7)
        << std::fixed << std::setprecision(3) << time << " s with counter = " << counter_[idx] << "\n";
    }
  }
  
  /**
   * @brief Print the average running time in each duration and how many times each duration has been recorded.
   * @note The unit of time is millisecond (ms), where 1s = 1000 ms.
   * This function should be called only once, after the loop.
   */
  void printTimeMilliseconds() const {
    for (int idx = 0; idx < num_stamp_; ++idx) {
      if (!counter_[idx]) continue;
      double time = sum_time_[idx].count() / 1000000.0 / counter_[idx];
      std::cout << "[LoopTimer] stamp" << std::setw(3) << idx << ":  " << std::setw(7)
        << std::fixed << std::setprecision(3) << time << " ms with counter = " << counter_[idx] << "\n";
    }
  }

  /**
   * @brief Print the average running time in each duration and how many times each duration has been recorded.
   * @note The unit of time is microsecond (us), where 1s = 1000000 us.
   * This function should be called only once, after the loop.
   */
  void printTimeMicroseconds() const {
    for (int idx = 0; idx < num_stamp_; ++idx) {
      if (!counter_[idx]) continue;
      double time = sum_time_[idx].count() / 1000.0 / counter_[idx];
      std::cout << "[LoopTimer] stamp" << std::setw(3) << idx << ":  " << std::setw(7)
        << std::fixed << std::setprecision(3) << time << " us with counter = " << counter_[idx] << "\n";
    }
  }

 private:
  int num_stamp_;  ///< The number of time stamps to be placed in the loop/function.

  std::chrono::steady_clock::time_point tic_;  ///< The startpoint of a time duration.

  std::chrono::steady_clock::time_point toc_;  ///< The endpoint of a time duration.

  std::vector<int> counter_;  ///< Container to keep how many times a designated time duration has repeated.

  /**
   * @brief Container to keep the sum of running time for each designated time duration.
   * @note chrono::nanoseconds is using signed long long int to store time;
   * 9,223,372,036,854,775,807 nanoseconds is equivalent to 106,752 days;
   * it will work fine as long as the total running time does not exceed this limit :)
   */
  std::vector<std::chrono::nanoseconds> sum_time_;
};

}  // namespace my
```


# Google Tools


# GLog

### Installation in Ubuntu

```bash
sudo apt install libgoogle-glog-dev
```

### Usage in CMake

Google GLog team does provide `GlogConfig.cmake` file, but it is not adopted by the default apt package available in Ubuntu. You can

* compile glog from source and enable CMake support, so that config file will be installed; or
* use default Ubuntu package by `sudo apt install libgoogle-glog-dev`, and then find a `FindGlog.cmake` file and add the path to this file to `CMAKE_MODULE_PATH` variable, such that CMake will use this file to load glog.

After the package is found, use the following variables for header files and libraries.&#x20;

```bash
find_package(Glog REQUIRED)  # GlogConfig.cmake NOT available; use FindGlog.cmake
include_directories(${GLOG_INCLUDE_DIRS})

add_executable (main src/main.cpp)
target_link_libraries (main ${GLOG_LIBRARIES})
```

#### References

* [Google Glog CMake docs](https://github.com/google/glog#cmake)
* [FindGlog.cmake in slambook2](https://github.com/gaoxiang12/slambook2/blob/master/ch13/cmake_modules/FindGlog.cmake)

### Tutorial / User Guide

Google glog defines a series of macros that simplify many common logging tasks. It provides logging APIs based on C++-style streams and various helper macros. You can log messages by severity level, control logging behavior from the command line, log based on conditionals, abort the program when expected conditions are not met, introduce your own verbose logging levels, and more.

You can log a message by simply streaming things to `LOG(<severity level>)`, e.g.,

```cpp
#include <glog/logging.h>

int main(int argc, char* argv[]) {
  // Initialize Google's logging library.
  google::InitGoogleLogging(argv[0]);

  // ...
  LOG(INFO) << "Found " << num_cookies << " cookies";
}
```

Unless otherwise specified, glog writes to the filename

* `/tmp/<program_name>.<hostname>.<username>.log.<severity_level>.<date>.<time>.<pid>`, or
* `/tmp/test.hteng-nuc.hteng.log.INFO.20210806-211849` for example.

Note that if `InitGoogleLogging` is not called for initialization, glog will turn into `logtostderr` mode and this log file will not be generated under `/tmp`. ([corresponding source code](https://github.com/google/glog/blob/master/src/logging.cc#L1791))

#### Severity Level

You can specify one of the following severity levels (in increasing order of severity): `INFO`, `WARNING`, `ERROR`, and `FATAL`.&#x20;

* Logging a `FATAL` message terminates the program (after the message is logged).
* Messages of a given severity are logged not only in the logfile for that severity, but also in all logfiles of lower severity. For example, a message of severity `FATAL` will be logged to the logfiles of severity `FATAL`, `ERROR`, `WARNING`, and `INFO`.
* The `DFATAL` severity logs a `FATAL` error in debug mode (i.e., there is no `NDEBUG` macro defined), but avoids halting the program in production by automatically reducing the severity to `ERROR`.
* By default, glog copies the log messages of severity level `ERROR` or `FATAL` to standard error (stderr) in addition to log files.

See the official [User Guide](https://github.com/google/glog#user-guide) for more tutorials.


# GFlags

### Installation in Ubuntu

```bash
sudo apt install libgflags-dev
```

### Usage in CMake

```bash
# gflags
find_package(GFlags REQUIRED)  # gflags-config.cmake available
include_directories(${GFLAGS_INCLUDE_DIR}) # not DIRS

add_executable (main src/main.cpp)
target_link_libraries (main ${GFLAGS_LIBRARIES})
```

### Tutorial

* [GFlags Documentation](https://gflags.github.io/gflags/) ([CMake section](https://gflags.github.io/gflags/#cmake))
* All DEFINE macros take the same three arguments: the name of the flag, its default value, and a 'help' string that describes its use. The 'help' string is displayed when the user runs the application with the `--help` flag.

```cpp
#include <gflags/gflags.h>
DEFINE_string(config, "./config/default.yaml", "config file path");
int main(int argc, char **argv) {
    google::ParseCommandLineFlags(&argc, &argv, true);
    myslam::VisualOdometry::Ptr vo(new myslam::VisualOdometry(FLAGS_config));
    ...
}

// in visual_odometry.h
VisualOdometry(std::string &config_path);
```


# GTest

### Installation in Ubuntu

```bash
sudo apt install libgtest-dev
```

### Usage in CMake

```bash
# gtest
find_package(GTest REQUIRED)  # GTestConfig.cmake available
include_directories(${GTEST_INCLUDE_DIRS})

add_executable (main src/main.cpp)
target_link_libraries (main ${GTEST_BOTH_LIBRARIES})

# alternatively
include(GoogleTest) # for CMake version > 3.9
```

### Tutorial

* [GoogleTest User’s Guide](https://google.github.io/googletest/) ([CMake section](https://google.github.io/googletest/quickstart-cmake.html))
* `ASSERT_*` versions generate fatal failures when they fail, and **abort the current function**.&#x20;
* `EXPECT_*` versions generate nonfatal failures, which don’t abort the current function.
* Test results will show up when you run the GTest executable after compilation.&#x20;

```cpp
// A simple test.
TEST(TestSuiteName, TestName) {//names must be valid C++ identifiers w/o underscore
  ... test body ...
}

// Tests factorial of 0.
TEST(FactorialTest, HandlesZeroInput) {
  EXPECT_EQ(Factorial(0), 1);
}

// Tests factorial of positive numbers.
TEST(FactorialTest, HandlesPositiveInput) {
  EXPECT_EQ(Factorial(1), 1);
  EXPECT_EQ(Factorial(2), 2);
  EXPECT_EQ(Factorial(3), 6);
  EXPECT_EQ(Factorial(8), 40320);
}

// Invoking the Tests
#include <gtest/gtest.h>
int main(int argc, char** argv) {
  ::testing::InitGoogleTest(&argc, argv);
  return RUN_ALL_TESTS(); // must not ignore the return value
}
```

### GMock

When you write a prototype or test, often it’s not feasible or wise to rely on real objects entirely. A mock object implements the same interface as a real object, but lets you specify at run time how it will be used and what it should do.

GMock has been merged into GTest project in 2019.&#x20;

* [gMock for Dummies](https://google.github.io/googletest/gmock_for_dummies.html)
* Github Repository: <https://github.com/google/googlemock>


# Style Guide

Google programming style has been widely adopted in software development.&#x20;

* [google](https://github.com/google)/[styleguide](https://github.com/google/styleguide)
* [Google C++ Style Guide](https://google.github.io/styleguide/cppguide.html)
* [zh-google-styleguide](https://github.com/zh-google-styleguide)/[zh-google-styleguide](https://github.com/zh-google-styleguide/zh-google-styleguide)
* [Google C++ Style Guide in Chinese](https://google-styleguide.readthedocs.io/zh_CN/latest/google-cpp-styleguide/contents.html)

## Summary (for C++)

### Header Files

* Be self-contained
* Avoid forward declaration
* The format of the symbol name in `#define` guard: `<PROJECT>_<PATH>_<FILE>_H_`
* Names and order of includes
  * Related header (the header that the current `.cc` file plans to implement or test)
  * C system headers
  * C++ standard library headers
  * Other libraries' .h files
  * Your project's .h files
  * Conditional includes

```cpp
#include "foo/server/fooserver.h"  // e.g., the current file is fooserver.cc

#include <sys/types.h>
#include <unistd.h>

#include <string>
#include <vector>

#include "base/basictypes.h"
#include "base/commandlineflags.h"
#include "foo/server/bar.h"

#ifdef LANG_CXX11
#include <initializer_list>
#endif  // LANG_CXX11
```

### Scoping

* With few exceptions, place code in a namespace.&#x20;
* Namespaces should have unique names based on the project name, and possibly its path.&#x20;
* **Do not use using-directives (e.g., using namespace foo). -- This pollutes the namespace!**
* Do not use inline namespaces.&#x20;
* Do not use namespace aliases at namespace scope **in header files**.

### Class

* Use a `struct` only for passive objects that carry data; everything else is a `class`.
* Use a `struct` instead of a pair or a tuple whenever the elements can have meaningful names.
* All inheritance should be `public`.
* Overload operators judiciously. Do not use user-defined literals.
* Define operators only on your own types.
* Make classes' data members `private`, unless they are constants.&#x20;

### Functions

* Non-optional input parameters should usually be values or `const` references.
* Use overloaded functions (including constructors) only if a reader can easily get it.
* Default arguments are allowed on non-virtual functions when the default is guaranteed to always have the same value.&#x20;
* Use trailing return types only where using the ordinary syntax (leading return types) is impractical or much less readable.

### Google-Specific Magic

* Prefer to have single, fixed owners for dynamically allocated objects.&#x20;
* Prefer to transfer ownership with smart pointers.
* Use [cpplint.py](https://github.com/google/styleguide/blob/gh-pages/cpplint/cpplint.py) to detect style errors.

### Other C++ Features

* We do not use C++ exceptions.
* Avoid using run-time type information (RTTI). (by use of `typeid` or `dynamic_cast`.)
* Use C++-style casts like `static_cast<float>(double_value)`. Do not use cast formats like `(int)x` unless the cast is to `void`. You may use cast formats like `T(x)` only when `T` is a class type.&#x20;
* Use the prefix form (`++i`) of the increment and decrement operators unless you need postfix semantics. (There can be a performance difference in certain cases.)
* We strongly recommend using `const` in APIs wherever it is meaningful and accurate.&#x20;
* You should not use the **unsigned** integer types such as `uint32_t`, unless there is a valid reason such as representing a bit pattern rather than a number. When in doubt, use a larger type.
* Avoid defining **macros**, especially in headers; prefer inline functions, enums, and const variables.&#x20;
* Use `nullptr` for pointers, and `'\0'` for chars (and not the `0` literal).
* Prefer `sizeof(varname)` to `sizeof(type)`.
* Use type deduction only if it makes the code clearer to **readers who aren't familiar with the project**.
* Use lambda expressions where appropriate. Prefer **explicit captures** when the lambda will escape the current scope.

### Naming

* Use names that describe the purpose or intent of the object. Do not worry about saving horizontal space as it is far more important to make your code immediately understandable by a new reader.
* **Filenames** should be all lowercase and can include underscores (`_`) or dashes (`-`).
* **Type names** start with a capital letter and have a capital letter for each new word, with no underscores: `MyExcitingClass`, `MyExcitingEnum`.
* The names of **variables** (including function parameters) and data members are all lowercase, with underscores between words. Data members of classes (but not structs) additionally have trailing underscores. For instance: `a_local_variable`, `a_struct_data_member`, `a_class_data_member_`.
* Variables declared **`constexpr`** or **`const`**, and whose value is fixed for the duration of the program, are named with a leading "k" followed by mixed case. Underscores can be used as separators in the rare cases where capitalization cannot be used for separation.
* Regular **functions** have mixed case; accessors and mutators may be named like variables.
* **Namespace names** are all lower-case, with words separated by underscores.
* **Enumerators** (for both scoped and unscoped enums) should be named like constants, not like macros. That is, use `kEnumName` not `ENUM_NAME`.
* Exceptions: If you are naming something that is analogous to an existing C or C++ entity then you can follow the existing naming convention scheme.

```cpp
// Type Names
class UrlTable { ...             // classes
struct UrlTableProperties { ...  // structs
typedef hash_map<UrlTableProperties *, std::string> PropertiesMap; // typedefs
using PropertiesMap = hash_map<UrlTableProperties *, std::string>; // using aliases
enum class UrlTableError { ...   // enums

// Variable Names
std::string table_name;  // common variable names
class TableInfo {        
 private:
  std::string table_name_;  // class data member - underscore at end
  static Pool<TableInfo>* pool_;
};
struct UrlTableProperties {
  std::string name;         // struct data member - no underscore
  static Pool<UrlTableProperties>* pool;
};

// Const Names
const int kDaysInAWeek = 7;
const int kAndroid8_0_0 = 24;  // Android 8.0.0

// Function Names
AddTableEntry()
DeleteUrl()
OpenFileOrDie()

// Enumerator Names
enum class UrlTableError {
  kOk = 0,
  kOutOfMemory,
  kMalformedInput,
};

// Exceptions
bigopen() // function name, follows form of open()
uint      // typedef
bigpos    // struct or class, follows form of pos
sparse_hash_map  // STL-like entity; follows STL naming conventions
LONGLONG_MAX     // a constant, as in INT_MAX
```

### Comments

* Use either the `//` or `/* */` syntax, as long as you are consistent.
* Start each file with license boilerplate.
* Every non-obvious **class or struct** declaration should have an accompanying comment that describes what it is for and how it should be used.
* Declaration comments describe use of the **function** (when it is non-obvious); comments at the definition of a function describe operation.
* In general the actual name of the variable should be descriptive enough to give a good idea of what the variable is used for. In certain cases, more comments are required.
* Note: I personally follows [the Doxygen style](https://www.doxygen.nl/manual/starting.html) for comments. Also, `@` is preferred over `\` for Doxygen commands.

```cpp
/**
 * @brief Abstract base class for Foo.
 */
class Foo {
 public:
  /**
   * @brief Constructor
   *
   * @param node configuration, which must contain one param: mem_var of double type
   */
  Foo(const Yaml::Node& node); 
  
 private:
  const double mem_var_;  ///< inline comments, 2 spaces before slash
};
```

### Formatting

* Each line of text in your code should be at most 80 characters long. (controversial)
* Non-ASCII characters should be rare, and must use UTF-8 formatting.
* Use only spaces, and indent 2 spaces at a time.
* See others by examples below.

```cpp
// Function Declarations and Definitions
ReturnType ClassName::FunctionName(Type par_name1, Type par_name2) {
  DoSomething();
}
ReturnType ClassName::ReallyLongFunctionName(Type par_name1, Type par_name2,
                                             Type par_name3) {
  DoSomething();
}
ReturnType LongClassName::ReallyReallyReallyLongFunctionName(
    Type par_name1,  // 4 space indent
    Type par_name2,
    Type par_name3) {
  DoSomething();  // 2 space indent
}

// Function Calls
bool result = DoSomething(argument1, argument2);
bool result = DoSomething(argument1, argument2,
                          argument3);  // wrap at parentheses
bool result = DoSomething(
    argument1, argument2, argument3);  // new line with 4 space indent

// Floating-point Literals
float f = 1.0f;  // bad: float f = 1.f;
float f2 = 1;    // Also OK  
long double ld = -0.5L;  // bad: long double ld = -.5L;
double d = 1248.0e6;     // bad: double d = 1248e6;

// Conditionals
if (condition) {     // no spaces inside parentheses, space before brace
  DoOneThing();      // two space indent
  DoAnotherThing();
} else if (int a = f(); a != 3) { // closing brace on new line, else on same line
  DoAThirdThing(a);
} else {
  DoNothing();
}
if (x == kFoo) return new Foo();  // allow one exception
if (x == kQuz) { return new Quz(1, 2, 3); } // a space between closing parenthesis

// Loops and Switch Statements
switch (var) {
  case 0: {  // 2 space indent
    ...      // 4 space indent
    break;
  }
  default: {
    assert(false);
  }
}

// Pointer and Reference Expressions
// These are fine, space preceding.
char *c;
const std::string &str;
int *GetPointer();
std::vector<char *>
// These are fine, space following (or elided).
char* c;
const std::string& str;
int* GetPointer();
std::vector<char*>  // Note no space between '*' and '>'

// Boolean Expressions
if (this_one_thing > this_other_thing &&
    a_third_thing == a_fourth_thing &&
    yet_another && last_one) {
  ...  // be consistent: && is always at the end of the lines
}

// Return Values
return result;  // No parentheses in the simple case.
// Parentheses OK to make a complex expression more readable.
return (some_long_condition &&
        another_condition);
// Do not needlessly surround the return expression with parentheses.
// Use parentheses in return expr; only where you would use them in x = expr;.

// Preprocessor Directives
  if (lopsided_score) {
#if DISASTER_PENDING      // Correct -- Starts at beginning of line
    DropEverything();
#endif
    BackToNormal();
  }

// Class Format
// Sections in public, protected and private order, each indented one space.
class MyClass : public OtherClass {
 public:      // Note the 1 space indent!  
  MyClass();  // Regular 2 space indent.
  explicit MyClass(int var);
};

// Constructor Initializer Lists
MyClass::MyClass(int var) : some_var_(var) {
  DoSomething();   // When everything fits on one line
}
MyClass::MyClass(int var)  
    : some_var_(var), some_other_var_(var + 1) {
  DoSomething();   // wrap before the colon and indent 4 spaces
}
MyClass::MyClass(int var)
    : some_var_(var),             // 4 space indent
      some_other_var_(var + 1) {  // lined up
  DoSomething();
}
MyClass::MyClass(int var)
    : some_var_(var) {}

// Namespace Formatting
namespace {

void foo() {  // Correct.  No extra indentation within namespace.
  ...
}

}  // namespace
```


# Clang Format

### Installation in Ubuntu

```shell
sudo apt install --no-install-recommends clang clang-format
```

### Set up Git Hook

* copy file `git-clang-format` to `/usr/local/bin`&#x20;
* copy file `pre-commit` to `.git/hooks` of the repository under development
* generate `.clang-format` file in the root directory of the project

#### File: pre-commit

<pre class="language-python"><code class="lang-python"><strong>#!/usr/bin/env python
</strong>
import sys
import subprocess
output = subprocess.check_output(["git", "clang-format", "--diff"])

if output not in ['no modified files to format\n', 'clang-format did not modify any files\n']:
  sys.exit("""Error: clang format check failed. Run git clang-format -f to fix it, then commit again.
NOTE: if (and *only* if) this is a merge commit, you may skip this check with 'git commit -n'""")
</code></pre>

#### File: git-clang-format

* <https://github.com/hanzheteng/LOCUS/blob/main/scripts/clang_setup/clang/git-clang-format>

#### File: .clang-format

* Example of full configuration: <https://github.com/hanzheteng/LOCUS/blob/main/.clang-format>
* A brief example: <https://github.com/MIT-SPARK/TEASER-plusplus/blob/master/.clang-format>

### Usage

* If git hook is set, run the following command before every git commit.

```shell
git clang-format -f
```

* In addition to being used as a git hook, clang can format a single c++ file in command line.

```shell
clang-format -i -style=google *.cpp  # use google default coding style
clang-format -i -style=file *.cpp  # if .clang-format is ready (in the same dir)
```

* To disable clang formatting in certain cases.

```cpp
int formatted_code;
// clang-format off
    void unformatted_code;
// clang-format on
void formatted_code_again;
```

### References

#### Official documentation

* <https://clang.llvm.org/docs/ClangFormat.html>
* <https://clang.llvm.org/docs/ClangFormatStyleOptions.html>

#### Tutorials

* <https://www.csd.uoc.gr/~hy255/tuts/2021/clang_format_tutorial.pdf>

#### A script to automatically install clang-format and set up git hook

* <https://github.com/hanzheteng/LOCUS/blob/main/scripts/clang_setup/clang_setup.sh>


# PCL


# Point Type

Header: [pcl/point\_types.h](https://github.com/PointCloudLibrary/pcl/blob/master/common/include/pcl/point_types.h)\
Implementation: [pcl/impl/point\_types.hpp](https://github.com/PointCloudLibrary/pcl/blob/master/common/include/pcl/impl/point_types.hpp)

### Intensity

* 4 bytes
* Members: float intensity
* intensity = 0.0f;

### Label

* 4 bytes
* Members: uint32\_t label

### RGB

* 4 bytes
* Members: uint32\_t rgba
* r = g = b = 0; a = 255;

### PointXY

* 8 bytes
* Members: float x, y

### **PointXYZ**

* 16 bytes (12 bytes used)
* Members: float x, y, z
* data\[3] = 1.0f;

### PointXYZI

* 32 bytes (12 bytes + 4 bytes used)
* Members: float x, y, z; float intensity
* data\[3] = 1.0f;
* intensity = 0.0f;

### PointXYZL

* 32 bytes (12 bytes + 4 bytes used)
* Members: float x, y, z; uint32\_t label
* data\[3] = 1.0f;
* label = 0;

### PointXYZRGB

* 32 bytes (12 bytes + 4 bytes used)
* Members: float x, y, z; uint32\_t rgba
* data\[3] = 1.0f;&#x20;
* r = g = b = 0;  &#x20;a = 255;

### PointXYZRGBA

* 32 bytes (12 bytes + 4 bytes used)&#x20;
* Members: float x, y, z; uint32\_t rgba
* data\[3] = 1.0f;&#x20;
* r = g = b = 0;  &#x20;a = 255;

### PointXYZRGBL

* 32 bytes (12 bytes + 8 bytes used)
* Members: float x, y, z; uint32\_t rgba, label
* data\[3] = 1.0f;&#x20;
* r = g = b = 0;  &#x20;a = 255;
* label = 0;

### Normal

* 32 bytes (12 bytes + 4 bytes used)
* Members: float normal\[3]; float curvature
* data\_n\[3] = 0.0f;&#x20;
* curvature = 0;

### PointWithScale

* 32 bytes (12 bytes + 16 bytes used)
* Members: float x, y, z; float scale, angle, response, octave
* data\[3] = 1.0f;
* scale = 1.0f;  &#x20;
* angle = -1.0f;  &#x20;
* response = 0.0f;  &#x20;
* octave = 0;

### PointNormal

* 48 bytes (12 bytes + 12 bytes + 4 bytes used)
* Members: float x, y, z; float normal\[3]; float curvature
* data\[3] = 1.0f;
* data\_n\[3] = 0.0f;&#x20;
* curvature = 0;

### PointXYZRGBNormal

* 48 bytes (12 bytes + 12 bytes + 8 bytes used)
* Members: float x, y, z; float normal\[3]; uint32\_t rgba, float curvature
* data\[3] = 1.0f;
* data\_n\[3] = 0.0f;&#x20;
* r = g = b = 0;  &#x20;a = 255;
* curvature = 0;

### PointSurfel

* 48 bytes (12 bytes + 12 bytes + 16 bytes used)
* Members: float x, y, z; float normal\[3]; uint32\_t rgba, float radius, confidence, curvature
* data\[3] = 1.0f;
* data\_n\[3] = 0.0f;&#x20;
* r = g = b = 0;  &#x20;a = 255;
* radius = confidence = curvature = 0.0f;


# Methods

## Keypoint

### ISS

* Input
  * PointXYZ
  * PointXYZRGBA   model
* Output
  * PointXYZ
  * PointXYZRGBA    model\_keypoints
* Notes
  * ISS + FPFH performs better than SIFT + FPFH
  * However, the registration can slip on the plane; there is no constraint on the tangent plane
  * `PointOutT p;  p.getVector3fMap () = ...`
* Algorithm
  * It computes an **unnormalized** covariance matrix **centered at the query point** (instead of the actual centroid/mean of all neighbors) and performs eigenvalue decomposition
  * The original algorithm proposed by the author assigns weights to each point inversely related to the point density (i.e. higher weight for sparse points), but in PCL the current implementation does not account for this (all points weighted equally).
  * The scatter matrix here is the **second-order mixed moment about the origin** (二阶混合原点矩).

### SIFT

* Input
  * \<typename PointT>    return p.intensity;
  * PointNormal        return p.curvature;
  * PointXYZRGB      return (static\_cast\<float> (299\*p.r + 587\*p.g + 114\*p.b) / 1000.0f);
  * PointXYZRGBA   return (static\_cast\<float> (299\*p.r + 587\*p.g + 114\*p.b) / 1000.0f);
  * Can be customized to use other types/fields ([example](https://github.com/PointCloudLibrary/pcl/blob/master/examples/keypoints/example_sift_z_keypoint_estimation.cpp#L51))
* Output
  * PointWithScale (save scale info if the `scale` field exists)
  * PointXYZ  (otherwise just provide the points)
* Notes
  * This method does not populate the keypoints indices --> there is no getKeypointsIndices()
* Algorithm
  * Similar to the 2D SIFT algorithm, it performs local minimum or maximum extraction in scale space for each octave (i.e. voxels of doublings of resolution), and performs the difference of Gaussian operation between each octave. If the local max/min meets the minimum contrast requirement, take it as a key point. (IMHO, it is kind of complicated, and not sure if it still makes sense in 3D space.)

### Harris3D

* Input
  * PointXYZ
  * PointXYZRGB
* Output
  * PointXYZI  (`intensity` field required)
* Notes
  * Need to set the threshold for intensity response for better performance
    * the function setThreshold();
* Algorithm
  * It computes intensity response for each and every point in the input cloud, according to five different rules (Harris, Noble, Lowe, Tomasi, Curvature), and performs non-max suppression to select those salient ones (also checks if intensity > threshold). Therefore, it is necessary to set the threshold at this step for better performance.&#x20;

## Feature

### Normal Estimation

* template \<typename PointInT, typename PointOutT>
* Input
  * PointXYZ
  * PointXYZRGBNormal
* Output
  * Normal
  * PointNormal
  * PointXYZRGBNormal
* Notes
  * estimates local surface properties (surface normals and curvatures) at each 3D point.
  * If PointOutT is specified as pcl::Normal, the normal is stored in the first 3 components (0-2), and the curvature is stored in component 3.
  * RadiusSearch and KSearch cannot be used at the same time --> pick only one approach
* Algorithm
  * The algorithm calls `computeCovarianceMatrix` function in centroid.h to compute **normalized** covariance matrix **centered at its actual centroid/mean** of all neighbors (not the query point).
  * Then it performs eigenvalue decomposition on this covariance matrix and extracts **the smallest** eigenvalue and its eigenvector, and takes this eigenvector as the normal vector.&#x20;
  * The curvature is computed by abs(smallest eigenvalue / trace of covariance matrix).
  * The covariance matrix is the **second-order central mixed moment** (二阶混合中心距).
  * For a plane, the estimated normal could have two opposite directions. The algorithm in PCL takes into consideration the view point of the point cloud and flips the estimated normal if cos(viewpoint - point, estimated normal) < 0. (cf. the method used in SHOT descriptor: check which half plane contains more points)

### FPFH

* template \<typename PointInT, typename PointNT, typename PointOutT = pcl::FPFHSignature33>
* Input
  * PointXYZ/PointXYZRGB + PointNormal/Normal
  * PointXYZRGBNormal + PointXYZRGBNormal&#x20;
* Output
  * FPFHSignature33
* Notes
  * The radius used here has to be larger than the radius used to estimate the surface normals!

## Registration

* setMaxCorrespondenceDistance()
  * Set the maximum distance threshold between two correspondent points in source <-> target.

    If the distance is larger than this threshold, the points will be ignored in the alignment process.

### ICP

* template \<typename PointSource, typename PointTarget, typename Scalar = float>
* The classic ICP registration algorithm using SVD (i.e. transformationEstimationSVD) is a wrap-up of the Eigen algorithm [Eigen::umeyama](https://eigen.tuxfamily.org/dox/group__Geometry__Module.html#gab3f5a82a24490b936f8694cf8fef8e60).

### SAC-IA

* setMinSampleDistance()
  * Set the minimum distances between samples.
  * Select s sample points from P while making sure that    &#x20;their pairwise distances are greater than a user-defined    &#x20;minimum distance dmin. (from Rusu FPFH ICRA2009 paper)

### NDT

* This algorithm runs slow?
* There is an issue in the computation of covariance in the NDT source code. This issue has not been fixed until PCL 1.12.0. The correct computation should multiply the term `n/(n-1)` but the code was incorrectly multiplying `(n-1)/n`.

```cpp
// pcl/filters/impl/voxel_grid_covariance.hpp

// line 329 and 330, in PCL 1.7, the wrong one
leaf.cov_ = (leaf.cov_ - 2 * (pt_sum * leaf.mean_.transpose ())) / leaf.nr_points + leaf.mean_ * leaf.mean_.transpose ();
leaf.cov_ *= (leaf.nr_points - 1.0) / leaf.nr_points;

// line 326 in PCL 1.12, the correct one
leaf.cov_ = (leaf.cov_ - pt_sum * leaf.mean_.transpose()) / (leaf.nr_points - 1.0);
```


# Architecture

## PCLBase

* virtual void **setInputCloud** (const PointCloudConstPtr \&cloud);
* virtual void **setIndices** (const IndicesConstPtr \&indices);
* bool **initCompute** ();    \[protected]
* bool **deinitCompute** ();    \[protected]

### Filter : public PCLBase

* :: void removeNaNFromPointCloud (...)
* inline void **filter** (PointCloud \&output)
* virtual void **applyFilter** (PointCloud \&output) = 0;    \[protected]

#### VoxelGrid : public Filter    ([voxel\_grid.h](https://pointclouds.org/documentation/classpcl_1_1_voxel_grid.html))

* inline void **setLeafSize** (float lx, float ly, float lz) / (const Eigen::Vector4f \&leaf\_size)
* inline void setDownsampleAllData (bool downsample)
* inline void setMinimumPointsNumberPerVoxel (unsigned int min\_points\_per\_voxel)
* inline void setFilterLimits (const double \&limit\_min, const double \&limit\_max)
* void **applyFilter** (PointCloud \&output);    \[protected]

### Keypoint : public PCLBase

* virtual void **setSearchSurface** (const PointCloudInConstPtr \&cloud) { surface\_ = cloud; }
* inline void **setSearchMethod** (const KdTreePtr \&tree) { tree\_ = tree; }
* inline void **setKSearch** (int k) { k\_ = k; }
* inline void **setRadiusSearch** (double radius) { search\_radius\_ = radius; }
* inline void **compute** (PointCloudOut \&output);
* inline int searchForNeighbors (...)
* virtual void **detectKeypoints** (PointCloudOut \&output) = 0;    \[protected]

#### HarrisKeypoint3D : public Keypoint    ([harris\_3d.h](https://pointclouds.org/documentation/classpcl_1_1_harris_keypoint3_d.html))

* void setMethod (ResponseMethod type);
* void setRadius (float radius);
* void **setThreshold** (float threshold);
* void **setNonMaxSupression** (bool = false);
* inline void setNumberOfThreads (unsigned int nr\_threads = 0) { threads\_= nr\_threads; }
* pcl::PointIndicesConstPtr  &#x20;**getKeypointsIndices** () { return (keypoints\_indices\_); }&#x20;

#### SIFTKeypoint : public Keypoint    ([sift\_keypoint.h](https://pointclouds.org/documentation/classpcl_1_1_s_i_f_t_keypoint.html))

* void **setScales** (float min\_scale, int nr\_octaves, int nr\_scales\_per\_octave);
* void **setMinimumContrast** (float min\_contrast);

#### ISSKeypoint3D : public Keypoint    ([iss\_3d.h](https://pointclouds.org/documentation/classpcl_1_1_i_s_s_keypoint3_d.html))

* void **setSalientRadius** (double salient\_radius)      \[ref: 6 \* model\_resolution]
* void **setNonMaxRadius** (double non\_max\_radius)     \[ref: 4 \* model\_resolution]
* void setNormalRadius (double normal\_radius)  &#x20;
* void setBorderRadius (double border\_radius)
* void **setMinNeighbors** (int min\_neighbors)        \[ref: 5]
* void setNormals (const PointCloudNConstPtr \&normals)
* void setNumberOfThreads (unsigned int nr\_threads=0)
* pcl::PointIndicesConstPtr  &#x20;**getKeypointsIndices** () { return (keypoints\_indices\_); }&#x20;

### Feature : public PCLBase

* inline void  &#x20;**setSearchSurface** (const PointCloudInConstPtr \&cloud)
* inline void **setSearchMethod** (const KdTreePtr \&tree) { tree\_ = tree; }
* inline void **setKSearch** (int k) { k\_ = k; }
* inline void **setRadiusSearch** (double radius)
* void  &#x20;**compute** (PointCloudOut \&output);
* inline int searchForNeighbors(...)    \[protected]
* virtual void **computeFeature** (PointCloudOut \&output) = 0;    \[private]

#### NormalEstimation : public Feature    ([normal\_3d.h](https://pointclouds.org/documentation/classpcl_1_1_normal_estimation.html))

* void **computeFeature** (PointCloudOut \&output);    \[protected]

#### FeatureFromNormals : public Feature

* inline void **setInputNormals** (const PointCloudNConstPtr \&normals) { normals\_ = normals; }

#### PFHEstimation : public FeatureFromNormals    ([pfh.h](https://pointclouds.org/documentation/classpcl_1_1_p_f_h_estimation.html))

#### FPFHEstimation : public FeatureFromNormals    ([fpfh.h](https://pointclouds.org/documentation/classpcl_1_1_f_p_f_h_estimation.html))

* void **computeFeature** (PointCloudOut \&output);    \[protected]

#### PFHRGBEstimation : public FeatureFromNormals    ([pfhrgb.h](https://pointclouds.org/documentation/classpcl_1_1_p_f_h_r_g_b_estimation.html))

#### SHOTEstimation : public FeatureFromNormals, FeatureWithLocalReferenceFrames   ([shot.h](https://pointclouds.org/documentation/classpcl_1_1_s_h_o_t_estimation.html))

#### SHOTColorEstimation : public FeatureFromNormals, FeatureWithLocalReferenceFrames    ([shot.h](https://pointclouds.org/documentation/classpcl_1_1_s_h_o_t_color_estimation.html))

### Registration : public PCLBase

* void  &#x20;setTransformationEstimation (const TransformationEstimationPtr \&te)
* void  &#x20;setCorrespondenceEstimation (const CorrespondenceEstimationPtr \&ce)
* virtual void **setInputSource** (const PointCloudSourceConstPtr \&cloud)
* virtual inline void&#x20;  **setInputTarget** (const PointCloudTargetConstPtr \&cloud)
* inline void setSearchMethodSource (const KdTreeReciprocalPtr \&tree, bool force\_no\_recompute = ...)
* inline void setSearchMethodTarget (const KdTreePtr \&tree, bool force\_no\_recompute = false)
* inline Matrix4  &#x20;**getFinalTransformation** () { return (final\_transformation\_); }
* inline Matrix4 getLastIncrementalTransformation () { return (transformation\_); }
* inline void&#x20;  **setMaxCorrespondenceDistance** (double distance\_threshold)       \[ref: 0.05]
* inline void&#x20;  **setMaximumIterations** (int nr\_iterations)          \[termination criterion 1; ref: 50, 500]
* inline void **setTransformationEpsilon** (double epsilon)      \[termination criterion 2; ref: 1e-6, 1e-8]
* inline void **setEuclideanFitnessEpsilon** (double epsilon)   \[termination criterion 3; ref: 1]
* inline void setRANSACIterations (int ransac\_iterations)
* inline void setRANSACOutlierRejectionThreshold (double inlier\_threshold)
* inline void **align** (PointCloudSource \&output);
* inline void addCorrespondenceRejector (const CorrespondenceRejectorPtr \&rejector)
* virtual void **computeTransformation** (PointCloudSource \&output, const Matrix4& guess)    \[protected]
* inline bool searchForNeighbors (const PointCloudSource &, int, std::vector &, std::vector &)\[protected]

#### SampleConsensusInitialAlignment : public Registration    ([ia\_ransac.h](https://pointclouds.org/documentation/classpcl_1_1_sample_consensus_initial_alignment.html))

* void **setSourceFeatures** (const FeatureCloudConstPtr \&features);
* void **setTargetFeatures** (const FeatureCloudConstPtr \&features);
* void&#x20;  **setMinSampleDistance** (float min\_sample\_distance)
* void&#x20;  setNumberOfSamples (int nr\_samples) { nr\_samples\_ = nr\_samples; }
* void  &#x20;setCorrespondenceRandomness (int k) { k\_correspondences\_ = k; }
* void  &#x20;setErrorFunction (const boost::shared\_ptr\<ErrorFunctor> & error\_functor)

#### FPCSInitialAlignment : public Registration    ([ia\_fpcs.h](https://pointclouds.org/documentation/classpcl_1_1registration_1_1_f_p_c_s_initial_alignment.html))

#### KFPCSInitialAlignment : public FPCSInitialAlignment   ([ia\_kfpcs.h](https://pointclouds.org/documentation/classpcl_1_1registration_1_1_k_f_p_c_s_initial_alignment.html))

#### NormalDistributionsTransform : public Registration      ([ndt.h](https://pointclouds.org/documentation/classpcl_1_1_normal_distributions_transform.html))

* inline void setResolution (float resolution)
* inline void setStepSize (double step\_size)
* inline void setOulierRatio (double outlier\_ratio)

#### IterativeClosestPoint : public Registration     ([icp.h](https://pointclouds.org/documentation/classpcl_1_1_iterative_closest_point.html))

* inline void setUseReciprocalCorrespondences (bool use\_reciprocal\_correspondence)
* IterativeClosestPointWithNormals : public IterativeClosestPoint       (icp.h)
* IterativeClosestPointNonLinear  &#x20;: public IterativeClosestPoint      (icp\_nl.h)

#### GeneralizedIterativeClosestPoint : public IterativeClosestPoint    ([gicp.h](https://pointclouds.org/documentation/classpcl_1_1_generalized_iterative_closest_point.html))

#### GeneralizedIterativeClosestPoint6D : public GeneralizedIterativeClosestPoint   ([gicp6d.h](https://pointclouds.org/documentation/gicp6d_8h_source.html))

### CorrespondenceEstimationBase : public PCLBase

* inline void  &#x20;**setInput**Source (const PointCloudSourceConstPtr& cloud)
* inline void  &#x20;setInputTarget (const PointCloudTargetConstPtr& cloud)
* inline void  &#x20;**setIndices**Source (const IndicesPtr& indices)
* inline void  &#x20;setIndicesTarget (const IndicesPtr& indices)
* inline void  &#x20;**setSearchMethod**Source(const KdTreeReciprocalPtr& tree,  &#x20;bool force\_no\_recompute = ...)
* inline void  &#x20;setSearchMethodTarget(const KdTreePtr& tree, bool force\_no\_recompute = false)
* virtual void  &#x20;**determineCorrespondences**(  pcl::Correspondences& correspondences,  &#x20;...) = 0;
* virtual void  &#x20;determineReciprocalCorrespondences(  pcl::Correspondences& correspondences,  &#x20;...) = 0;
* inline void  &#x20;setPointRepresentation(const PointRepresentationConstPtr& point\_representation)

#### CorrespondenceEstimation&#xD; : public CorrespondenceEstimationBase   （[corres\_est.h](https://pointclouds.org/documentation/classpcl_1_1registration_1_1_correspondence_estimation.html)）

* void  &#x20;**determineCorrespondences** (  pcl::Correspondences& correspondences,  &#x20;...) override;
* void  &#x20;determineReciprocalCorrespondences(  pcl::Correspondences& correspondences,  &#x20;...) override;

## Registration::TransformationEstimation

* virtual void **estimateRigidTransformation** (const pcl::PointCloud< PointSource > \&cloud\_src, const pcl::PointCloud< PointTarget > \&cloud\_tgt, Matrix4 \&transformation\_matrix) const =0

#### TransformationEstimationSVD&#xD; : public TransformationEstimation

* virtual void  &#x20;getTransformationFromCorrelation(...) const     \[protected]

#### TransformationEstimationLM&#xD; : public TransformationEstimation    ([trans\_est.h](https://pointclouds.org/documentation/classpcl_1_1registration_1_1_transformation_estimation_l_m.html))

* virtual MatScalar  &#x20;**computeDistance**(const Vector4& p\_src, const PointTarget& p\_tgt)    \[protected]
* struct OptimizationFunctor : public Functor      \[protected]

#### TransformationEstimationPointToPlane&#xD; : public TransformationEstimationLM

* Scalar  &#x20;computeDistance(const Vector4& p\_src, const PointTarget& p\_tgt) const override   \[protected]

## Search

* virtual void setSortedResults (bool sorted);
* virtual void **setInputCloud** (const PointCloudConstPtr& cloud, const IndicesConstPtr \&indices = ...);
* virtual int nearestKSearch (...)&#x20;
* virtual int **radiusSearch** (...)
* void sortResults (std::vector& indices, std::vector& distances) const;   \[protected]

#### KdTree : public Search    ([search/kdtree.h](https://pointclouds.org/documentation/classpcl_1_1search_1_1_kd_tree.html))

* template\<typename PointT, class Tree = pcl::KdTreeFLANN\<PointT> >
* void setPointRepresentation (const PointRepresentationConstPtr \&point\_representation);
* void setSortedResults (bool sorted\_results);
* void **setInputCloud** (const PointCloudConstPtr& cloud, const IndicesConstPtr& indices = ...);
* void setEpsilon (float eps);
* int nearestKSearch (...)
* int **radiusSearch** (...)

## Visualization

### PCLVisualizer  ([visualizer](https://pointclouds.org/documentation/classpcl_1_1visualization_1_1_p_c_l_visualizer.html))

* void setBackgroundColor (const double \&r, const double \&g, const double \&b, int viewport = 0);
* void createViewPort (double xmin, double ymin, double xmax, double ymax, int \&viewport)
* boost::signals2::connection **registerKeyboardCallback** (std::function...)
* boost::signals2::connection **registerMouseCallback** (std::function...)
* void initCameraParameters ()
* void setCameraParameters (const Eigen::Matrix3f \&intrinsics, const Eigen::Matrix4f \&extrinsics, int viewport=0)
* void **setCameraPosition** (double pos\_x, double pos\_y, double pos\_z, double view\_x, double view\_y, double view\_z, double up\_x, double up\_y, double up\_z, int viewport=0)
* bool setShapeRenderingProperties (int property, double value, const std::string \&id, int viewport=0)
* bool **addPointCloud** (...)
* bool addPointCloudNormals (...)
* bool addPointCloudPrincipalCurvatures (...)
* bool **addCorrespondences** (...)
* bool addPolygonMesh (...)
* bool addText (...)
* bool addText3D (...)
* bool wasStopped () const
* void spinOnce (int time=1, bool force\_redraw=false)

### PointCloudColorHandler  ([color\_handler](https://pointclouds.org/documentation/classpcl_1_1visualization_1_1_point_cloud_color_handler.html))

* PointCloudColorHandler (const PointCloudConstPtr \&cloud) :  &#x20;cloud\_ (cloud), capable\_ (false), field\_idx\_ (-1), fields\_ ()  &#x20;{}

#### PointCloudColorHandlerCustom : public PointCloudColorHandler

* PointCloudColorHandlerCustom (const PointCloudConstPtr \&cloud, double r, double g, double b)

#### PointCloudColorHandlerRGBField : public PointCloudColorHandler

#### PointCloudColorHandlerLabelField : public PointCloudColorHandler

### PointCloudGeometryHandler  ([geometry\_handler](https://pointclouds.org/documentation/classpcl_1_1visualization_1_1_point_cloud_geometry_handler.html))

#### PointCloudGeometryHandlerCustom : public PointCloudGeometryHandler

* Custom handler class for PointCloud geometry. Given an input dataset and three user defined fields, all data present in them is extracted and displayed on screen as XYZ data.&#x20;

### PCLHistogramVisualizer  ([histogram\_visualizer](https://pointclouds.org/documentation/classpcl_1_1visualization_1_1_p_c_l_histogram_visualizer.html))

* bool addFeatureHistogram (const pcl::PointCloud< PointT > \&cloud, int hsize, ...)

### PCLPlotter  ([plotter](https://pointclouds.org/documentation/classpcl_1_1visualization_1_1_p_c_l_plotter.html))

* void addPlotData (double const \*array\_X, double const \*array\_Y, unsigned long size, ...)
* void addHistogramData (std::vector< double > const \&data, int const nbins=10, ...)
* bool addFeatureHistogram (const pcl::PointCloud< PointT > \&cloud, int hsize, ...)
* void setBackgroundColor (const double r, const double g, const double b)
* void plot ()

## IO

### FileReader

#### PCDReader : public FileReader   ([pcd\_io.h](https://pointclouds.org/documentation/classpcl_1_1_p_c_d_reader.html))

* int loadPCDFile (const std::string \&file\_name, pcl::PointCloud< PointT > \&cloud)
  * {pcl::PCDReader p;  return (p.read (file\_name, cloud));}

#### PLYReader : public FileReader   ([ply\_io.h](https://pointclouds.org/documentation/classpcl_1_1_p_l_y_reader.html))

* int loadPLYFile (const std::string \&file\_name, pcl::PointCloud< PointT > \&cloud)


# Code Explained

## Basics

```cpp
std::vector<PointT, Eigen::aligned_allocator<PointT> > points;
```

* `pcl::PointCloud<PointT>` uses `std::vector` internally as the container to store points, and applies the aligned allocator that Eigen inherited from `std::allocator`.

```cpp
pcl::PointCloud<pcl::PointXYZ>::Ptr cloud (new pcl::PointCloud<pcl::PointXYZ>);
```

* `pcl::XX::Ptr` uses `shared_ptr` internally as the smart pointer to keep track of the number of references and automatically manage the memory. The `shared_ptr` is a class essentially.
* `pcl::XX::Ptr cloud (new XX)` takes the pointer created by `new` and passes to the constructor as an argument. The constructor also initializes other variables such as the one used to keep track of the number of references.&#x20;

```cpp
pcl::XX::Ptr cloud (new pcl::PointCloud<pcl::PointXYZ>);
pcl::XX::Ptr cloud (new pcl::PointCloud<pcl::PointXYZ>() );
```

* `new Thing();` is explicit that you want a constructor called whereas `new Thing;` is taken to imply you don't mind if the constructor isn't called. See [this reference page](https://stackoverflow.com/questions/620137/do-the-parentheses-after-the-type-name-make-a-difference-with-new) and [this example](https://docs.microsoft.com/en-us/cpp/cpp/how-to-create-and-use-shared-ptr-instances?view=msvc-160#example-1).
* For example, `new Thing();` may initialize all values to 0, but `new Thing;` may not.

```cpp
// correct
pcl::PointCloud<pcl::PointXYZRGB> source_cloud_transformed;
sac_ia.align (source_cloud_transformed);

// incorrect
pcl::PointCloud<pcl::PointXYZRGB>::Ptr source_cloud_transformed (new pcl::PointCloud<pcl::PointXYZRGB>);
sac_ia.align (source_cloud_transformed);

// because align function takes in only raw pointers rather than smart pointers
```

* In this example, `new` should not be used, since we want to leave the constructor to the `align` function. Otherwise the compilation error is the following.

> error: no matching function for call to ‘pcl::SampleConsensusInitialAlignment\<pcl::PointXYZRGB, pcl::PointXYZRGB, pcl::FPFHSignature33>::align(pcl::PointCloud\<pcl::PointXYZRGB>::Ptr&)’

> sac\_ia.align (source\_cloud\_transformed);

> no known conversion for argument 1 from ‘pcl::PointCloud\<pcl::PointXYZRGB>::Ptr {aka boost::shared\_ptr\<pcl::PointCloud\<pcl::PointXYZRGB> >}’ to ‘pcl::Registration\<pcl::PointXYZRGB, pcl::PointXYZRGB, float>::PointCloudSource& {aka pcl::PointCloud\<pcl::PointXYZRGB>&}’

## Correspondence Estimation & KdTreeFLANN

```cpp
// pcl/registration/correspondence_estimation.h
typename pcl::search::KdTree<PointTarget>::Ptr tree_;
CorrespondenceEstimation::determineCorrespondences ()
{
  tree_->nearestKSearch (input_->points[*idx], 1, index, distance);
}

// pcl/search/kdtree.h & pcl/search/impl/kdtree.hpp
template<typename PointT, class Tree = pcl::KdTreeFLANN<PointT> >
class KdTree: public Search<PointT>
boost::shared_ptr<Tree> tree_;
int nearestKSearch ()
{
  return (tree_->nearestKSearch (point, k, k_indices, k_sqr_distances));
}

// pcl/kdtree/kdtree_flann.h & pcl/kdtree/impl/kdtree_flann.hpp
template <typename PointT, typename Dist = ::flann::L2_Simple<float> >
class KdTreeFLANN : public pcl::KdTree<PointT>
typedef ::flann::Index<Dist> FLANNIndex;
boost::shared_ptr<FLANNIndex> flann_index_;
int nearestKSearch ()
{
  flann_index_->knnSearch (::flann::Matrix<float> (&query[0], 1, dim_), 
                           k_indices_mat, k_distances_mat,
                           k, param_k_);
}

// flann/flann.hpp
template<typename Distance>
class Index
typedef NNIndex<Distance> IndexType;
IndexType* nnIndex_;
int knnSearch()
{
  return nnIndex_->knnSearch(queries, indices, dists, knn, params);
}

// flann/algorithms/nn_index.h
template <typename Distance>
class NNIndex : public IndexBase
int knnSearch()
{
  // actual implementation using OpenMP
  // wrapped by functions and compilation directives
}
```

* The default precision used in FLANN is 0.8, according to the autotuned parameter settings [here](https://github.com/mariusmuja/flann/blob/master/src/cpp/flann/algorithms/autotuned_index.h#L57).&#x20;

## KdTree Point Representation

PCL uses the `PointRepresentation` class as a middle layer to convert various point types (including common ones as well as point types created for feature descriptors) into unified float vectors, to pass into the Kd-tree.&#x20;

For example, the PointXYZRGBA point type can have 3 float fields (12 bytes) for 3D position, and 4 unit8 fields (4 bytes) for RGBA color channels. However, KdTree can only accept uniform vectors with all fields to be float, and will measure distance equally in each dimension. Therefore, there is need to adapt the two interfaces. By default, it will only take the first 3 dimensions/fields in the point type (the 3D position of the point).

If there is need to incorporate more fields into the kdtree search, then we need to implement a customized `PointRepresentation` class for this purpose. For example for the XYZRGB point type, how to map from uint8 to float, how to normalize the range, and how to assign weights to distance measure need to be considered carefully. In addition, it might be good to implement a customized distance measure as needed, because by default the kdtree will use `L2_Simple` for input float vectors.

```cpp
// pcl/point_representation.h
template <typename PointT>
class PointRepresentation {
  protected:
    int nr_dimensions_;
    std::vector<float> alpha_;
    bool trivial_;
  public:
    PointRepresentation () : nr_dimensions_ (0), alpha_ (0), trivial_ (false) {}
    virtual bool 
      isValid (const PointT &p) const
    template <typename OutputType> void 
      vectorize (const PointT &p, OutputType &out) const
};

// extends PointRepresentation to define default behavior for common point types.
template <typename PointDefault>
class DefaultPointRepresentation : public PointRepresentation <PointDefault> {
  DefaultPointRepresentation () {
    nr_dimensions_ = sizeof (PointDefault) / sizeof (float);
    if (nr_dimensions_ > 3) nr_dimensions_ = 3;
    trivial_ = true;
  }
};

// intended to be used when defining the default behavior for feature descriptor
// types (i.e., copy each element of each field into a float array).
template <typename PointDefault>
class DefaultFeatureRepresentation : public PointRepresentation <PointDefault>

// extends PointRepresentation to allow for sub-part selection on the point.
template <typename PointDefault>
class CustomPointRepresentation : public PointRepresentation <PointDefault>
```

In order to use customized point representation, we need to replace `point_representation_` in KdTreeFLANN with our customized one. By default it is set to `DefaultPointRepresentation` initialized in its parent KdTree class.&#x20;

```cpp
// pcl/search/kdtree.h
template<typename PointT, class Tree = pcl::KdTreeFLANN<PointT> >
class KdTree: public Search<PointT>

// pcl/kdtree/kdtree_flann.h
template <typename PointT, typename Dist = ::flann::L2_Simple<float> >
class KdTreeFLANN : public pcl::KdTree<PointT> {
  using KdTree<PointT>::point_representation_;
};

// pcl/kdtree/kdtree.h
template <typename PointT>
class KdTree {
  KdTree (bool sorted = true) : input_(), indices_(), 
    epsilon_(0.0f), min_pts_(1), sorted_(sorted), 
    point_representation_ (new DefaultPointRepresentation<PointT>) {};
};
```

## KdTree with Customized Distance Measure

By default, PCL has only compiled `L2_Simple` as the distance measure, and set a header guard `PCL_NO_PRECOMPILE` to avoid the recompilation of the hpp files.

```cpp
// in file pcl/search/kdtree.h
// ...
#ifdef PCL_NO_PRECOMPILE
#include <pcl/search/impl/kdtree.hpp>  // will be skipped by default
#else
#define PCL_INSTANTIATE_KdTree(T) template class PCL_EXPORTS pcl::search::KdTree<T>;
#endif
```

In this case, using distance measure other than the default `L2_Simple` will cause the following error.&#x20;

```cpp
undefined reference to `pcl::search::KdTree<pcl::PointXYZRGBNormal, pcl::KdTreeFLANN<pcl::PointXYZRGBNormal, flann::L1<float> > >::KdTree(bool)'
```

The solution is to enforce the compilation of the hpp file again.

```cpp
// to use pcl::Search::KdTree with L1 distance
#include <pcl/search/impl/kdtree.hpp>
pcl::search::KdTree<PointT, pcl::KdTreeFLANN<PointT, flann::L1<float>>>::Ptr kdtree
  (new pcl::search::KdTree<PointT, pcl::KdTreeFLANN<PointT, flann::L1<float>>>);

// to use pcl::KdTreeFLANN with L1 distance
#include <pcl/kdtree/impl/kdtree_flann.hpp>
boost::shared_ptr<pcl::KdTreeFLANN<FeatureT, flann::L1<float> > > feature_tree 
  (new pcl::KdTreeFLANN<FeatureT, flann::L1<float> >);
```

## Point Cloud Transformation

```cpp
pcl::transformPointCloudWithNormals (*source_cloud, *source_cloud_transformed, transformation);
```

Recall that one of the use cases of transformation matrices is to change the reference frame. Suppose that point $$p$$ is fixed in space. Given $$p\_1$$and $$p\_2$$in frame 1 and frame 2 respectively, we have the relation $$p\_2 = T\_{21} p\_1$$, where $$T\_{21}$$is the transformation matrix between frame 1 and frame 2 (in this case, indicating the configuration of frame 1 with respect to frame 2).

Note that when we transform a point cloud, we actually apply the transformation matrix to each and every point in the cloud (not the origin, not the reference frame). Therefore, it follows the aforementioned relation $$p\_2 = T\_{21} p\_1$$.&#x20;

Accordingly in PCL, when we use `pcl::transformPointCloud` , the transformation matrix being used is actually $$T\_{21}$$(not $$T\_{12}$$), though with this matrix we can obtain the new point cloud 2 from point cloud 1. **It is important to keep in mind this subscript, especially when subscript cancellation is needed in the following steps.**


# Open3D


# Python API

### I/O

```python
# read and write
cloud1 = o3d.io.read_point_cloud("cloud1.pcd")
cloud2 = o3d.io.read_point_cloud("cloud2.ply")
o3d.io.write_point_cloud("cloud3.pcd", cloud3)
```

### Geometry

```python
# use np.asarray() to access data
cloud_ndarray = np.asarray(cloud.points)

# use o3d.utility.Vector3dVector to convert to o3d format
cloud.points = o3d.utility.Vector3dVector(cloud_ndarray)

# can also change data using np.asarray
np.asarray(cloud.colors)[idx[1:], :] = [0, 1, 0] # color nbr points to green

# paint color
cloud.paint_uniform_color([1.0, 0.0, 0.0])

# K-D Tree and NN search
cloud_tree = o3d.geometry.KDTreeFlann(cloud)
[k, idx, _] = cloud_tree.search_knn_vector_3d(cloud.points[1500], 200)
[k, idx, dist] = cloud_tree.search_radius_vector_3d(query_point, radius)
```

### Visualization

```python
# basic viewers at high level
o3d.visualization.draw_geometries([pcd])
o3d.visualization.draw_geometries_with_editing([pcd])

# custom visualization
def custom_draw_geometry(pcd):
    # The following code achieves the same effect as:
    # o3d.visualization.draw_geometries([pcd])
    vis = o3d.visualization.Visualizer()
    vis.create_window()
    vis.add_geometry(pcd)
    vis.run()
    vis.destroy_window()

# non-blocking visualization
vis = o3d.visualization.Visualizer()
vis.create_window()
for i in range(icp_iteration):
    # do ICP single iteration
    # transform geometry using ICP
    vis.update_geometry(geometry)
    vis.poll_events()
    vis.update_renderer()
```


# Registration

### Color ICP

```python
#!/usr/bin/env python3
import os
import copy
import numpy as np
import open3d as o3d

def draw_registration_result(source, target, transformation, color=False):
    source_temp = copy.deepcopy(source)
    target_temp = copy.deepcopy(target)
    if not color:
        source_temp.paint_uniform_color([1, 1, 0])
        target_temp.paint_uniform_color([0, 1, 1])
    source_temp.transform(transformation)
    o3d.visualization.draw_geometries([source_temp, target_temp])

def icp_registration(source, target, max_corres_dist):
    print("ICP registration")
    result = o3d.pipelines.registration.registration_icp(source, target, max_corres_dist)
    print(result)
    print(result.transformation)
    draw_registration_result(source, target, result.transformation, color=False)

def colored_icp_registration(source, target, voxel_size):
    print("Colored ICP registration")
    voxel_radius = [5*voxel_size, 3*voxel_size, voxel_size]
    max_iter = [60, 35, 20]
    current_transformation = np.identity(4)
    for scale in range(3):
        max_it = max_iter[scale]
        radius = voxel_radius[scale]
        print("scale_level = {0}, voxel_size = {1}, max_iter = {2}".format(scale, radius, max_it))
        source_down = source.voxel_down_sample(radius)
        target_down = target.voxel_down_sample(radius)
        source_down.estimate_normals(o3d.geometry.KDTreeSearchParamHybrid(radius=radius*2, max_nn=20))
        target_down.estimate_normals(o3d.geometry.KDTreeSearchParamHybrid(radius=radius*2, max_nn=20))
        result = o3d.pipelines.registration.registration_colored_icp(
            source_down, 
            target_down, 
            radius, 
            current_transformation,
            o3d.pipelines.registration.TransformationEstimationForColoredICP(),
            o3d.pipelines.registration.ICPConvergenceCriteria(relative_fitness=1e-6,
                                                              relative_rmse=1e-6,
                                                              max_iteration=max_it))
        current_transformation = result.transformation
        print(result)
    print(current_transformation)
    draw_registration_result(source, target, current_transformation, color=False)

if __name__ == "__main__":
    # unset env variable to re-enable OpenGL 3.3 in VM environments
    if 'SVGA_VGPU10' in os.environ:
        del os.environ['SVGA_VGPU10']

    # parameters
    voxel_size = 0.01
    max_corres_dist = 3*voxel_size
    transformation = np.identity(4)

    # load point clouds
    source = o3d.io.read_point_cloud("../data/rgb_redwood_lobby/1.pcd")
    target = o3d.io.read_point_cloud("../data/rgb_redwood_lobby/2.pcd")
    print("Loaded " + str(len(source.points)) + " points for source point cloud")
    print("Loaded " + str(len(target.points)) + " points for target point cloud")
    
    # estimate normals
    source.estimate_normals(search_param=o3d.geometry.KDTreeSearchParamHybrid(radius=2*voxel_size, max_nn=16))
    target.estimate_normals(search_param=o3d.geometry.KDTreeSearchParamHybrid(radius=2*voxel_size, max_nn=16))
    print("Estimated normals for source and target point clouds")

    # show initial alignment
    print("Initial alignment")
    print(o3d.pipelines.registration.evaluate_registration(source, target, max_corres_dist))
    draw_registration_result(source, target, transformation, color=False)

    # ICP
    icp_registration(source, target, max_corres_dist)

    # colored ICP
    colored_icp_registration(source, target, voxel_size)
```


# Visualization

### PCD Viewer

```python
#!/usr/bin/env python3
import sys
import open3d as o3d

if __name__ == "__main__":
    pcd = o3d.io.read_point_cloud(sys.argv[1])
    o3d.visualization.draw_geometries([pcd])
```

### Keypoint Viewer

```python
#!/usr/bin/env python3
import os
import open3d as o3d

def keypoints_to_spheres(keypoints):
    spheres = o3d.geometry.TriangleMesh()
    for keypoint in keypoints.points:
        sphere = o3d.geometry.TriangleMesh.create_sphere(radius=0.01)
        sphere.translate(keypoint)
        spheres += sphere
    spheres.paint_uniform_color([1.0, 0.0, 0.0])
    return spheres

if __name__ == "__main__":
    os.system('../build/test_keypoint ../config/params.yaml')
    cloud = o3d.io.read_point_cloud("../data/cloud.pcd")
    keypoints = o3d.io.read_point_cloud("../data/keypoint.pcd")
    o3d.visualization.draw_geometries([cloud, keypoints_to_spheres(keypoints)])
```

### Crop Polygon

```python
#!/usr/bin/env python3
import sys
import open3d as o3d

if __name__ == "__main__":
    print("Demo for manual geometry cropping")
    print("1) Press 'Y' twice to align geometry with negative direction of y-axis")
    print("2) Press 'K' to lock screen and to switch to selection mode")
    print("3) Drag for rectangle selection,")
    print("   or use ctrl + left click for polygon selection")
    print("4) Press 'C' to get a selected geometry and to save it")
    print("5) Press 'F' to switch to freeview mode")
    pcd = o3d.io.read_point_cloud(sys.argv[1])
    o3d.visualization.draw_geometries_with_editing([pcd])
```

### Crop Selected Sphere

```python
#!/usr/bin/env python3
import os
import numpy as np
import open3d as o3d


def pick_points(cloud):
    print("Press [shift + left click] to pick a point")
    print("Press [shift + right click] to undo point picking")
    print("Press 'Q' to close the window after picking points")
    vis = o3d.visualization.VisualizerWithEditing()
    vis.create_window()
    vis.add_geometry(cloud)
    vis.run()  # user picks points
    vis.destroy_window()
    return vis.get_picked_points() # a list of indices of picked points


if __name__ == "__main__":
    if 'SVGA_VGPU10' in os.environ:
        del os.environ['SVGA_VGPU10']
    cloud = o3d.io.read_point_cloud("../data/cloud.pcd")
    save_to_file = "../data/cropped.pcd"
    search_radius = 0.05
    
    # pick a point in interactive visualization
    point_indices = pick_points(cloud)
    picked_point = cloud.points[point_indices[0]] # take the first point
    print("picked point = " + str(picked_point))

    # find all neighbor points around the picked point
    cloud_tree = o3d.geometry.KDTreeFlann(cloud)
    [k, idx, _] = cloud_tree.search_radius_vector_3d(picked_point, search_radius)

    # crop the spherical region and save to a new file
    cropped = o3d.geometry.PointCloud()
    cropped.points = o3d.utility.Vector3dVector(np.asarray(cloud.points)[idx])
    cropped.colors = o3d.utility.Vector3dVector(np.asarray(cloud.colors)[idx])
    o3d.io.write_point_cloud(save_to_file, cropped)
    print("save the cropped point cloud to " + save_to_file)

    # color the spherical region to green to confirm
    np.asarray(cloud.colors)[idx[1:], :] = [0, 1, 0]
    o3d.visualization.draw_geometries([cloud])

    # show the cropped sphere
    sphere = o3d.io.read_point_cloud("../data/cropped.pcd")
    o3d.visualization.draw_geometries([sphere])

```

### Keypoint Animation

```python
#!/usr/bin/env python3
import os
import time
import open3d as o3d


def viewer(cloud, keypoints):
    flag  = False
    idx = -1
    max_idx = len(keypoints)
    vis = o3d.visualization.VisualizerWithKeyCallback()
    vis.create_window()
    vis.add_geometry(cloud)

    def next_one(vis):
        nonlocal idx
        if idx >= 0 and idx < max_idx - 1:
            idx = idx + 1
            vis.remove_geometry(keypoints[idx - 1], False)
            vis.add_geometry(keypoints[idx], False) # False: keep current viewpoint
            print("current keypoint = " + str(keypoints[idx].get_center()))
        elif idx == -1:
            idx = idx + 1
            vis.add_geometry(keypoints[idx], False)
            print("current keypoint = " + str(keypoints[idx].get_center()))
        elif idx == max_idx - 1:
            idx = idx + 1
            vis.remove_geometry(keypoints[idx - 1], False)
            print("current keypoint = no keypoint selected")

    def previous_one(vis):
        nonlocal idx
        if idx > 0 and idx < max_idx:
            idx = idx - 1
            vis.remove_geometry(keypoints[idx + 1], False)
            vis.add_geometry(keypoints[idx], False) # False: keep current viewpoint
            print("current keypoint = " + str(keypoints[idx].get_center()))
        elif idx == max_idx:
            idx = idx - 1
            vis.add_geometry(keypoints[idx], False)
            print("current keypoint = " + str(keypoints[idx].get_center()))
        elif idx == 0:
            idx = idx - 1
            vis.remove_geometry(keypoints[idx + 1], False)
            print("current keypoint = no keypoint selected")

    def forward_animation(vis):
        nonlocal flag
        if flag:
            next_one(vis)
            time.sleep(0.5)
 
    def pause_or_resume(vis):
        nonlocal flag
        if flag == False:
            flag = True
        else:
            flag = False

    # ord(X) turns X into unicode; need to use capital letter here
    vis.register_key_callback(ord("."), next_one)
    vis.register_key_callback(ord(","), previous_one)
    vis.register_key_callback(ord(" "), pause_or_resume)
    # animation callback will get run repeatability
    vis.register_animation_callback(forward_animation)
    vis.run()
    vis.destroy_window()


def keypoints_to_spheres(keypoints, r):
    spheres = []
    for keypoint in keypoints.points:
        sphere = o3d.geometry.TriangleMesh.create_sphere(radius=r)
        sphere.translate(keypoint)
        sphere.paint_uniform_color([1.0, 0.0, 0.0])
        spheres.append(sphere)
    return spheres


if __name__ == "__main__":
    if 'SVGA_VGPU10' in os.environ:
        del os.environ['SVGA_VGPU10']
    print("Demo for visualizing keypoints")
    print("Press '.' to view the next keypoint")
    print("Press ',' to view the previous keypoint")
    print("Press space key to start the animation")
    print("Press space key again to pause/resume")
    cloud = o3d.io.read_point_cloud("../data/cloud.pcd")
    keypoints = o3d.io.read_point_cloud("../data/keypoint.pcd")
    viewer(cloud, keypoints_to_spheres(keypoints, 0.04))
```

### References

* [Interactive visualization tutorial](http://www.open3d.org/docs/latest/tutorial/visualization/non_blocking_visualization.html)
* [Non-blocking visualization tutorial](http://www.open3d.org/docs/latest/tutorial/Advanced/non_blocking_visualization.html)
* [Python code examples for visualization](https://github.com/isl-org/Open3D/tree/master/examples/python/visualization)


# Tools

### bag2pcd

```bash
sudo apt install pcl_ros
roscore
rosrun pcl_ros bag_to_pcd <input_file.bag> <topic> <output_directory>
```

### bag2vicon

* In one terminal `rosbag play <input_file.bag>`&#x20;
* In another terminal run `./bag2vicon.py`. It will subscribe the designated ros topic and save it into a txt file in the format \[time, trans.x, trans.y, trans.z, rot.x, rot.y, rot.z, rot.w].
* Remember to change ros topic name, bag name, file name accordingly beforehand.&#x20;

```python
#!/usr/bin/env python
import csv
import rospy
from geometry_msgs.msg import TransformStamped

pose = []

def callback(msg):
    secs = msg.header.stamp.secs
    nsecs = msg.header.stamp.nsecs
    time = float(str(secs) + "." + str(nsecs))
    trans = msg.transform.translation
    rot = msg.transform.rotation
    stamp_float = [time, trans.x, trans.y, trans.z, rot.x, rot.y, rot.z, rot.w]
    stamp_string = [format(s, ".4f") for s in stamp_float ]
    pose.append(stamp_string)

if __name__ == '__main__':
    ros_topic_name = "vicon/turtlebot/turtlebot"
    ros_bag_name = "turtlebot_zed_pointcloud_sq2_vicon.bag"
    save_to_file = "turtlebot_zed_pointcloud_sq2_vicon_gt.txt"
    try:
        rospy.init_node('converter', anonymous=True)
        rospy.Subscriber(ros_topic_name, TransformStamped, callback)
        rospy.spin()
    finally:
        with open(save_to_file, mode='w') as file:
            file.writelines("# ground truth trajectory\n")
            file.writelines("# file: " + ros_bag_name + "\n")
            file.writelines("# timestamp tx ty tz qx qy qz qw\n")
        with open(save_to_file, mode='a') as file:
            file = csv.writer(file, delimiter=' ', quotechar='"', quoting=csv.QUOTE_MINIMAL)
            for line in pose:
                file.writerow(line)
```

### vicon2gt

* usage: `./vicon2gt.py <vicon_traj.txt> <pcd_folder>`
* It will iterate over pcd folder, extract the timestamp from the pcd filename, and look for the closest timestamp in `vicon_traj.txt` and save it into a `gt.log` file.&#x20;

```python
#!/usr/bin/python
import argparse
import random
import numpy
import sys
import os
_EPS = numpy.finfo(float).eps * 4.0

class CameraPose:
    def __init__(self, meta, mat):
        self.metadata = meta
        self.pose = mat
    def __str__(self):
        return 'Metadata : ' + ' '.join(map(str, self.metadata)) + '\n' + \
            "Pose : " + "\n" + numpy.array_str(self.pose)

def mr_read_trajectory(filename):
    traj = []
    with open(filename, 'r') as f:
        metastr = f.readline()
        while metastr:
            metadata = map(int, metastr.split())
            mat = numpy.zeros(shape = (4, 4))
            for i in range(4):
                matstr = f.readline()
                mat[i, :] = numpy.fromstring(matstr, dtype = float, sep=' \t')
            traj.append(CameraPose(metadata, mat))
            metastr = f.readline()
    return traj

def mr_write_trajectory(traj, filename):
    with open(filename, 'w') as f:
        for x in traj:
            p = x.pose.tolist()
            f.write(' '.join(map(str, x.metadata)) + '\n')
            f.write('\n'.join(' '.join(map('{0:.12f}'.format, p[i])) for i in range(4)))
            f.write('\n')

def transform44(l):
    """
    Generate a 4x4 homogeneous transformation matrix from a 3D point and unit quaternion.
    
    Input:
    l -- tuple consisting of (stamp,tx,ty,tz,qx,qy,qz,qw) where
         (tx,ty,tz) is the 3D position and (qx,qy,qz,qw) is the unit quaternion.
         
    Output:
    matrix -- 4x4 homogeneous transformation matrix
    """
    t = l[1:4]
    q = numpy.array(l[4:8], dtype=numpy.float64, copy=True)
    nq = numpy.dot(q, q)
    if nq < _EPS:
        return numpy.array((
        (                1.0,                 0.0,                 0.0, t[0])
        (                0.0,                 1.0,                 0.0, t[1])
        (                0.0,                 0.0,                 1.0, t[2])
        (                0.0,                 0.0,                 0.0, 1.0)
        ), dtype=numpy.float64)
    q *= numpy.sqrt(2.0 / nq)
    q = numpy.outer(q, q)
    return numpy.array((
        (1.0-q[1, 1]-q[2, 2],     q[0, 1]-q[2, 3],     q[0, 2]+q[1, 3], t[0]),
        (    q[0, 1]+q[2, 3], 1.0-q[0, 0]-q[2, 2],     q[1, 2]-q[0, 3], t[1]),
        (    q[0, 2]-q[1, 3],     q[1, 2]+q[0, 3], 1.0-q[0, 0]-q[1, 1], t[2]),
        (                0.0,                 0.0,                 0.0, 1.0)
        ), dtype=numpy.float64)

def read_trajectory(filename, matrix=True):
    """
    Read a trajectory from a text file. 
    
    Input:
    filename -- file to be read
    matrix -- convert poses to 4x4 matrices
    
    Output:
    dictionary of stamped 3D poses
    """
    file = open(filename)
    data = file.read()
    lines = data.replace(","," ").replace("\t"," ").split("\n") 
    list = [[float(v.strip()) for v in line.split(" ") if v.strip()!=""] for line in lines if len(line)>0 and line[0]!="#"]
    list_ok = []
    for i,l in enumerate(list):
        if l[4:8]==[0,0,0,0]:
            continue
        isnan = False
        for v in l:
            if numpy.isnan(v): 
                isnan = True
                break
        if isnan:
            sys.stderr.write("Warning: line %d of file '%s' has NaNs, skipping line\n"%(i,filename))
            continue
        list_ok.append(l)
    if matrix :
      traj = dict([(l[0],transform44(l[0:])) for l in list_ok])
    else:
      traj = dict([(l[0],l[1:8]) for l in list_ok])
    return traj

def find_closest_index(L,t):
    """
    Find the index of the closest value in a list.
    
    Input:
    L -- the list (sorted in ascending order)
    t -- value to be found
    
    Output:
    index of the closest element
    """
    beginning = 0
    difference = abs(L[0] - t)
    best = 0
    end = len(L)
    while beginning < end:
        middle = int((end+beginning)/2)
        if abs(L[middle] - t) < difference:
            difference = abs(L[middle] - t)
            best = middle
        if t == L[middle]:
            return middle
        elif L[middle] > t:
            end = middle
        else:
            beginning = middle + 1
    return best

if __name__ == '__main__':
    if len(sys.argv) < 3:
        print("usage: ./vicon2gt.py <vicon_traj.txt> <pcd_folder>")
        exit()

    vicon = read_trajectory(sys.argv[1])
    stamps_gt = list(vicon.keys())
    stamps_gt.sort()

    directory = os.listdir(sys.argv[2])
    stamps_file = list()
    for filename in directory:
        if filename.rsplit('.', 1)[1] == 'pcd':
            stamps_file.append(float(filename.rsplit('.', 1)[0]))
    stamps_file.sort()

    total_length = len(stamps_file)
    i = 0
    traj = []
    for stamp in stamps_file:
        idx = find_closest_index(stamps_gt, stamp)
        trans = vicon[stamps_gt[idx]]
        metadata = [i, i, total_length]
        traj.append(CameraPose(metadata, trans))
        i += 1
    mr_write_trajectory(traj, "gt.log")
```

### ply2pcd

```python
#!/usr/bin/env python3
import sys
import open3d as o3d

def ply2pcd(plyFileName, pcdFileName):
    pcd = o3d.io.read_point_cloud(plyFileName)
    o3d.io.write_point_cloud(pcdFileName, pcd)

if __name__ == "__main__":
    filename = sys.argv[1]
    if filename.rsplit('.', 1)[1] == 'ply':
        pcdname = filename.rsplit('.', 1)[0] + '.pcd'
        ply2pcd(filename, pcdname)
```

### ply2pcd batch

```python
#!/usr/bin/env python3
import sys
import os
import open3d as o3d

def ply2pcd(plyFileName, pcdFileName):
    pcd = o3d.io.read_point_cloud(plyFileName)
    o3d.io.write_point_cloud(pcdFileName, pcd)

if __name__ == "__main__":
    path = sys.argv[1]
    listdir = os.listdir(path)
    for filename in listdir:
        if filename.split('.')[1] == 'ply':
            pathin = path + '/' + filename
        pathout = path + '/' + filename.split('.')[0] + '.pcd'
        ply2pcd(pathin, pathout)
```

### pcd2ply

```python
#!/usr/bin/env python3
import sys
import open3d as o3d

def pcd2ply(pcdFileName, plyFileName):
    pcd = o3d.io.read_point_cloud(pcdFileName)
    o3d.io.write_point_cloud(plyFileName, pcd)

if __name__ == "__main__":
    filename = sys.argv[1]
    if filename.rsplit('.', 1)[1] == 'pcd':
        newname = filename.rsplit('.', 1)[0] + '.ply'
        pcd2ply(filename, newname)
```

### pcd2ply batch

```python
#!/usr/bin/env python3
import sys
import os
import open3d as o3d

def pcd2ply(pcdFileName, plyFileName):
    pcd = o3d.io.read_point_cloud(pcdFileName)
    o3d.io.write_point_cloud(plyFileName, pcd)

if __name__ == "__main__":
    path = sys.argv[1]
    listdir = os.listdir(path)
    for filename in listdir:
        if filename.split('.')[1] == 'pcd':
            pathin = path + '/' + filename
        pathout = path + '/' + filename.split('.')[0] + '.ply'
        pcd2ply(pathin, pathout)
```

### bin2pcd

```python
#!/usr/bin/env python3
import numpy as np
import struct
import sys
import open3d as o3d

def bin_to_pcd(binFileName):
    size_float = 4
    list_pcd = []
    with open(binFileName, "rb") as f:
        byte = f.read(size_float * 4)
        while byte:
            x, y, z, intensity = struct.unpack("ffff", byte)
            list_pcd.append([x, y, z])
            byte = f.read(size_float * 4)
    np_pcd = np.asarray(list_pcd)
    pcd = o3d.geometry.PointCloud()
    pcd.points = o3d.utility.Vector3dVector(np_pcd)
    return pcd

def main(binFileName, pcdFileName):
    pcd = bin_to_pcd(binFileName)
    o3d.io.write_point_cloud(pcdFileName, pcd)

if __name__ == "__main__":
    a = sys.argv[1]
    b = sys.argv[2]
    main(a, b)
```


# OpenCV


# Documentation

## Tutorials (v3.1.0)

### Introduction

* [Installation in Linux](https://docs.opencv.org/3.1.0/d7/d9f/tutorial_linux_install.html)
* [Using OpenCV with gcc and CMake](https://docs.opencv.org/3.1.0/db/df5/tutorial_linux_gcc_cmake.html)
* [Load and Display an Image](https://docs.opencv.org/3.1.0/db/deb/tutorial_display_image.html)
* [Load, Modify, and Save an Image](https://docs.opencv.org/3.1.0/db/d64/tutorial_load_save_image.html)

### The Core Functionality (core module)

* [Mat - The Basic Image Container](https://docs.opencv.org/3.1.0/d6/d6d/tutorial_mat_the_basic_image_container.html)
* [How to scan images, lookup tables and time measurement](https://docs.opencv.org/3.1.0/db/da5/tutorial_how_to_scan_images.html)
* [Basic Drawing](https://docs.opencv.org/3.1.0/d3/d96/tutorial_basic_geometric_drawing.html)
* [Mask operations on matrices](https://docs.opencv.org/3.1.0/d7/d37/tutorial_mat_mask_operations.html) (a.k.a. Kernel operations)
* [File Input and Output using XML and YAML files](https://docs.opencv.org/3.1.0/dd/d74/tutorial_file_input_output_with_xml_yml.html) (cv::FileStorage data structure)

### Image Processing (imgproc module)

* [Smoothing Images](https://docs.opencv.org/3.1.0/dc/dd3/tutorial_gausian_median_blur_bilateral_filter.html) (cv::GaussianBlur, cv::bilateralFilter)
* [Image Pyramids](https://docs.opencv.org/3.1.0/d4/d1f/tutorial_pyramids.html)
* [Sobel Derivatives](https://docs.opencv.org/3.1.0/d2/d2c/tutorial_sobel_derivatives.html)
* [Hough Circle Transform](https://docs.opencv.org/3.1.0/d4/d70/tutorial_hough_circle.html)
* [Image Moments](https://docs.opencv.org/3.1.0/d0/d49/tutorial_moments.html)

### Camera calibration and 3D reconstruction (calib3d module)

* [Camera calibration with square chessboard](https://docs.opencv.org/3.1.0/dc/d43/tutorial_camera_calibration_square_chess.html)
* [Camera calibration With OpenCV](https://docs.opencv.org/3.1.0/d4/d94/tutorial_camera_calibration.html)
  \*

  ```
  Determine the distortion matrix
  ```

  * Determine the camera matrix
  * Take input from Camera, Video and Image file list
  * Read configuration from XML/YAML file
  * Save the results into XML/YAML file
  * Calculate re-projection error
* [Real time pose estimation of a textured object](https://docs.opencv.org/3.1.0/dc/d2c/tutorial_real_time_pose.html)
  \*

  ```
  Read 3D textured object model and object mesh.
  ```

  * Take input from Camera or Video.
  * Extract ORB features and descriptors from the scene.
  * Match scene descriptors with model descriptors using Flann matcher.
  * Pose estimation using PnP + Ransac.
  * Linear Kalman Filter for bad poses rejection.

### 2D Features framework (feature2d module)

* [Harris corner detector](https://docs.opencv.org/3.1.0/d4/d7d/tutorial_harris_detector.html)
* [Creating your own corner detector](https://docs.opencv.org/3.1.0/d9/dbc/tutorial_generic_corner_detector.html)
* [Feature Detection](https://docs.opencv.org/3.1.0/d7/d66/tutorial_feature_detection.html)
* [Feature Description](https://docs.opencv.org/3.1.0/d5/dde/tutorial_feature_description.html)
  * [Feature Matching with FLANN](https://docs.opencv.org/3.1.0/d5/d6f/tutorial_feature_flann_matcher.html)
* [Features2D + Homography to find a known object](https://docs.opencv.org/3.1.0/d7/dff/tutorial_feature_homography.html) (transformation estimation)
* [AKAZE and ORB planar tracking](https://docs.opencv.org/3.1.0/dc/d16/tutorial_akaze_tracking.html)

### Tutorials for contrib modules

* [ArUco marker detection (aruco module)](https://docs.opencv.org/3.1.0/d9/d6d/tutorial_table_of_content_aruco.html)
* [Line Features Tutorial](https://docs.opencv.org/3.1.0/df/dfa/tutorial_line_descriptor_main.html)
* [Structure From Motion](https://docs.opencv.org/3.1.0/de/d7c/tutorial_table_of_content_sfm.html)


# Modules

## cv::Mat

* With C++ interface in OpenCV 2.0, Mat supports automatic memory management.
* Mat has two data parts: the matrix header (constant size) and a pointer to the matrix containing the pixel values (varying size).
* For computational efficiency, OpenCV uses a reference counting system.&#x20;
  * Each Mat object has its own header, however the matrix may be shared.
  * The copy operators will only copy the headers and the pointer to the large matrix, not the data itself.
  * Making a modification using any of them will affect all the other ones as well.
* To copy the matrix itself, use `cv::Mat::clone()` and `cv::Mat::copyTo()`
* Color space options
  * RGB is the most common one (however OpenCV uses BGR instead)
  * The HSV and HLS decompose colors into their hue, saturation and value/luminance components (more natural way for us to describe colors)
  * YCrCb is used by the popular JPEG image format
  * CIE L\*a\*b\* is a perceptually uniform color space

```cpp
// cv::Mat::Mat Constructor
Mat M(2, 2, CV_8UC3, Scalar(0,0,255));

// Use C/C++ arrays and initialize via constructor
int sz[3] = {2,2,2};
Mat L(3, sz, CV_8UC(1), Scalar::all(0));

// MATLAB style initializer
Mat E = Mat::eye(4, 4, CV_64F);
Mat O = Mat::ones(2, 2, CV_32F);
Mat Z = Mat::zeros(3, 3, CV_8UC1);

// For small matrices you may use comma separated initializers
Mat C = (Mat_<double>(3,3) << 0, -1, 0, -1, 5, -1, 0, -1, 0);

// Create a new header for an existing Mat object
Mat RowClone = C.row(1).clone();

// You can fill out a matrix with random values
Mat R = Mat(3, 2, CV_8UC3);
randu(R, Scalar::all(0), Scalar::all(255));

// Other common OpenCV data structures
Point2f P(5, 1);
Point3f P3f(2, 6, 7);
std::vector<float> v(20);  Mat(v);
std::vector<Point2f> vPoints(20);

// Typedefs in CV Core module
// typedef Mat_< Vec3f > 	cv::Mat3f
// typedef Matx< double, 3, 3 > 	cv::Matx33d
```

## cv::ORB

```cpp
cv::Mat image1 = cv::imread("../data/1.png", cv::IMREAD_COLOR);
cv::Mat image2 = cv::imread("../data/2.png", cv::IMREAD_COLOR);

cv::Ptr<cv::ORB> orb = cv::ORB::create();
std::vector<cv::KeyPoint> keypoints1, keypoints2;
orb->detect(image1, keypoints1);
orb->detect(image2, keypoints2);

cv::Mat descriptors1, descriptors2;
orb->compute(image1, keypoints1, descriptors1);
orb->compute(image2, keypoints2, descriptors2);

std::vector<cv::DMatch> matches;
cv::Ptr<cv::DescriptorMatcher> matcher = cv::DescriptorMatcher::create("BruteForce-Hamming");
matcher->match(descriptors1, descriptors2, matches, cv::Mat());

cv::Mat output_image;
cv::drawMatches(image1, keypoints1, image2, keypoints2, matches, output_image);
cv::namedWindow("Display window", cv::WINDOW_AUTOSIZE);
cv::imshow("Display window", output_image);
cv::waitKey(0);
```

## cv::FileStorage

```cpp
// XML/YAML File Open and Close
string filename = "I.xml";
FileStorage fs(filename, FileStorage::WRITE);

fs.open(filename, FileStorage::READ);

// Input and Output of text and numbers
fs << "iterationNr" << 100;

int itNr;
fs["iterationNr"] >> itNr;
itNr = (int) fs["iterationNr"];

// Input/Output of OpenCV Data structures
Mat R = Mat_<uchar >::eye  (3, 3),
    T = Mat_<double>::zeros(3, 1);
fs << "R" << R;    // Write cv::Mat
fs << "T" << T;
fs["R"] >> R;      // Read cv::Mat
fs["T"] >> T;

// Input/Output of vectors (arrays) and associative maps
// use cv::FileNode and cv::FileNodeIterator data structures

// Read and write your own data structures
// ...
```

## Line Features

* OpenCV contrib modules [Line Features Tutorial](https://docs.opencv.org/3.1.0/df/dfa/tutorial_line_descriptor_main.html)
* [cv::LineSegmentDetector](https://docs.opencv.org/3.1.0/db/d73/classcv_1_1LineSegmentDetector.html) implements the LSD detector.
* [cv::line\_descriptor](https://docs.opencv.org/3.1.0/dc/ddd/group__line__descriptor.html) provides the line descriptors as one of the contrib modules. Not sure if this implements the Line Band Descriptors (LBD) that is commonly used for line matching.


# Other Libraries


# Eigen

### Examples

```cpp
#include <Eigen/Core>
#include <Eigen/Dense>

using namespace std;
using namespace Eigen;

// declarations
Matrix<float, 2, 3> matrix_23; // type, row, col
Vector3d v_3d;      // Matrix<float, 3, 1>
Matrix3d matrix_33; // Matrix<double, 3, 3>
MatrixXd matrix_x;  // Matrix<double, Dynamic, Dynamic>

// input & output
matrix_33 = Matrix3d::Zero();
matrix_23 << 1, 2, 3, 4, 5, 6;
cout << "matrix 2x3 from 1 to 6: \n" << matrix_23 << endl;
cout << "matrix(1, 2): " << matrix_23(0, 1) << endl;
v_3d << 3, 2, 1;

// requires explicit type conversion (float to double)
Matrix<double, 2, 1> result = matrix_23.cast<double>() * v_3d;
cout << "[1,2,3;4,5,6]*[3,2,1]=" << result.transpose() << endl;

// matrix operations
matrix_33 = Matrix3d::Random();
cout << "random matrix: \n" << matrix_33 << endl;
cout << "transpose: \n" << matrix_33.transpose() << endl;
cout << "sum: " << matrix_33.sum() << endl;
cout << "trace: " << matrix_33.trace() << endl;
cout << "times 10: \n" << 10 * matrix_33 << endl;
cout << "inverse: \n" << matrix_33.inverse() << endl;
cout << "det: " << matrix_33.determinant() << endl;

// eigen value
SelfAdjointEigenSolver<Matrix3d> eigen_solver(matrix_33.transpose() * matrix_33);
cout << "Eigen values = \n" << eigen_solver.eigenvalues() << endl;
cout << "Eigen vectors = \n" << eigen_solver.eigenvectors() << endl;

// solve equations
Matrix<double, N, N> matrix_NN = MatrixXd::Random(N, N);
matrix_NN = matrix_NN * matrix_NN.transpose();  // for semi-definite
Matrix<double, N, 1> v_Nd = MatrixXd::Random(N, 1);
x = matrix_NN.colPivHouseholderQr().solve(v_Nd);  // QR decomposition
x = matrix_NN.ldlt().solve(v_Nd);  // cholesky decomposition

// timing
clock_t tic = clock();
cout << "time is " << 1000 * (clock() - tic) / (double) CLOCKS_PER_SEC << "ms" << endl;

```

### Linear Solver

We discuss some properties of the matrix $$A^TA$$, constructed from a m-by-n matrix $$A$$ where m is significantly greater than n. This is normally the case in least-square problems.

* The matrix $$A^TA$$ is always symmetric by definition.
* The matrix $$A^TA$$ is positive semi-definite if it is invertible, since we have$$x^TA^TAx=(Ax)^TAx=∥Ax∥^2≥0$$.
* The matrix $$A^TA$$  is almost always invertible, because m is significantly greater than n, and hence the constructed matrix is high likely to be full-rank.
* The dense linear solver `ldlt()` in Eigen is the fastest for positive or negative semi-definite matrices. Remember to use `double` precision to avoid numerical instability issues.

References: [Quora: Is ATA always positive definite?](https://qr.ae/pvKU3m); [StackExchange: Is a matrix multiplied with its transpose something special?](https://math.stackexchange.com/a/158225/884562); [StackExchange: Proof for why a matrix multiplied by its transpose is positive semidefinite](https://math.stackexchange.com/questions/1463140/proof-for-why-a-matrix-multiplied-by-its-transpose-is-positive-semidefinite); [Eigen Documentation: Basic linear solving](https://eigen.tuxfamily.org/dox/group__TutorialLinearAlgebra.html);&#x20;

### Numerical Stability

The following logs reveal one of the potential issues of using `A.ldlt().solve(b);`&#x20;

* The solver was used for Gauss-Newton optimization in an Iterative Closest Point algorithm. It was tested under the termination condition of max\_iteration only.&#x20;
* The result suggests that we should check if A and b are healthy before sending to `A.ldlt().solve(b);`, and it might be better to add epsilon as another termination condition.
* Note that `float` type supports a value range of 1.2E-38 to 3.4E+38, and 6-7 decimal digits, while `double` type supports a value range of 2.3E-308 to 1.7E+308, and 15 decimal digits.
* Updates: This issue was resolved by switching to `double` precision. (Tested multiple times towards 100 iterations. The produced result is stable.)

```
// Example Log 1
it = 0; cloud size = 29280; selected size = 7246
JTJ = 
 38016.2 -13125.2 -18423.8  6854.26 -1088.44  14895.9
-13125.2  6123.19  6490.22 -2408.94   365.29  -5207.7
-18423.8  6490.22   8958.4 -3330.11  527.498 -7219.55
 6854.26 -2408.94 -3330.11  1241.23 -196.889  2693.11
-1088.44   365.29  527.498 -196.889  31.7478 -427.549
 14895.9  -5207.7 -7219.55  2693.11 -427.549  5856.14
JTJ determinant = 9.95116e+07
JTr = 
 -577.63
 228.602
 281.135
-105.983
 16.6081
-230.903
X = 
  -0.112421
-0.00316341
  -0.102398
  -0.291014
   0.121551
   0.339041
transformation in this iteration
  0.994757   0.101927 0.00834039  -0.291014
 -0.102219   0.988446   0.111918   0.121551
 0.0031634  -0.112183   0.993683   0.339041
         0          0          0          1
current transformation estimation
  0.994757   0.101927 0.00834039  -0.291014
 -0.102219   0.988446   0.111918   0.121551
 0.0031634  -0.112183   0.993683   0.339041
         0          0          0          1
it = 1; cloud size = 29280; selected size = 24791
JTJ = 
 92950.4 -42927.1 -45647.5  19505.3 -2807.42  42299.9
-42927.1  27583.8  21442.2 -9525.14   1324.4 -20755.5
-45647.5  21442.2  22517.7 -9641.93  1382.54 -20829.7
 19505.3 -9525.14 -9641.93  4282.87 -598.361  9236.44
-2807.42   1324.4  1382.54 -598.361   90.107 -1298.59
 42299.9 -20755.5 -20829.7  9236.44 -1298.59  20018.1
JTJ determinant = 2.59903e+14
JTr = 
-138.758
 66.6306
  68.979
-33.9927
 3.96181
 -71.508
X = 
  0.0378539
-0.00402526
  0.0855123
   0.203131
   0.024896
 -0.0837208
transformation in this iteration
    0.996338   -0.0854987 -0.000775407     0.203131
   0.0854074     0.995619   -0.0380502     0.024896
  0.00402525    0.0378446     0.999276   -0.0837208
           0            0            0            1
current transformation estimation
   0.999851   0.0171299 -0.00202947  -0.0974733
 -0.0169317     0.99709   0.0743299    0.108159
 0.00329683  -0.0742844    0.997232    0.258503
          0           0           0           1
it = 2; cloud size = 29280; selected size = 27278
JTJ = 
  106650 -49865.1 -52593.7  21952.8  -3155.5  47461.4
-49865.1  32271.1  24970.3 -10820.4  1500.07 -23530.8
-52593.7  24970.3  26055.8 -10891.8  1559.69 -23452.8
 21952.8 -10820.4 -10891.8   4732.8 -661.023  10178.9
 -3155.5  1500.07  1559.69 -661.023  99.2445 -1430.34
 47461.4 -23530.8 -23452.8  10178.9 -1430.34    22006
JTJ determinant = 6.6466e+14
JTr = 
-69.9797
 44.7118
 35.9486
-17.6539
 1.80711
-36.3435
X = 
  0.0231947
-0.00401701
  0.0506327
    0.13484
  0.0331909
 -0.0589205
transformation in this iteration
    0.99871  -0.0506905 -0.00283697     0.13484
  0.0506107    0.998445  -0.0233662   0.0331909
   0.004017   0.0231924    0.999723  -0.0589205
          0           0           0           1
current transformation estimation
   0.999411  -0.0332245 -0.00862379    0.031276
  0.0336208    0.998142   0.0508101    0.130209
 0.00691963  -0.0510701    0.998671    0.201628
          0           0           0           1
it = 3; cloud size = 29280; selected size = 28816
JTJ = 
  117818 -54770.5 -58277.6  23776.2 -3413.26  51235.2
-54770.5  34908.8  27502.5 -11604.7   1608.9 -25130.5
-58277.6  27502.5  28969.2 -11832.1  1691.92 -25385.1
 23776.2 -11604.7 -11832.1  5025.81 -703.071  10773.6
-3413.26  1608.89  1691.92 -703.071  105.135 -1516.08
 51235.2 -25130.5 -25385.1  10773.6 -1516.08  23220.2
JTJ determinant = 1.12071e+15
JTr = 
-29.1478
 17.9882
 14.8802
  -7.199
0.732457
-14.8477
X = 
 0.00985068
-0.00153022
  0.0214971
  0.0555449
  0.0158326
 -0.0239884
transformation in this iteration
   0.999768  -0.0215095 -0.00131805   0.0555449
  0.0214954     0.99972 -0.00988113   0.0158326
 0.00153022  0.00985051     0.99995  -0.0239884
          0           0           0           1
current transformation estimation
  0.998446  -0.054619  -0.011031  0.0837472
 0.0550257   0.997653  0.0407425   0.144685
0.00877979 -0.0412862   0.999109    0.17896
         0          0          0          1
it = 4; cloud size = 29280; selected size = 29062
JTJ = 
 inf -nan -nan -inf -nan -nan
-nan  inf -inf -nan -inf -inf
-nan -inf  inf -nan  inf  inf
-inf -nan -nan  inf -nan -nan
-nan -inf  inf -nan  inf  inf
-nan -inf  inf -nan  inf  inf
JTJ determinant = -nan
JTr = 
 inf
-nan
-nan
-inf
-nan
-nan
X = 
-nan
-nan
-nan
-nan
-nan
   0
transformation in this iteration
-nan -nan -nan -nan
-nan -nan -nan -nan
-nan -nan -nan    0
   0    0    0    1
current transformation estimation
-nan -nan -nan -nan
-nan -nan -nan -nan
-nan -nan -nan -nan
   0    0    0    1
color_icp: /usr/include/pcl-1.8/pcl/kdtree/impl/kdtree_flann.hpp:136: int pcl::KdTreeFLANN<PointT, Dist>::nearestKSearch(const PointT&, int, std::vector<int>&, std::vector<float>&) const [with PointT = pcl::PointXYZRGBNormal; Dist = flann::L2_Simple<float>]: Assertion `point_representation_->isValid (point) && "Invalid (NaN, Inf) point coordinates given to nearestKSearch!"' failed.
Aborted (core dumped)
```

```
// Example Log 2
it = 0; cloud size = 29280; selected size = 7246
JTJ =  38016.2 -13125.2 -18423.8  6854.26 -1088.44  14895.9
-13125.2  6123.19  6490.22 -2408.94   365.29  -5207.7
-18423.8  6490.22   8958.4 -3330.11  527.498 -7219.55
 6854.26 -2408.94 -3330.11  1241.23 -196.889  2693.11
-1088.44   365.29  527.498 -196.889  31.7478 -427.549
 14895.9  -5207.7 -7219.55  2693.11 -427.549  5856.14
JTr =  -577.63
 228.602
 281.135
-105.983
 16.6081
-230.903
X =   -0.112421
-0.00316341
  -0.102398
  -0.291014
   0.121551
   0.339041
transformation in this iteration
  0.994757   0.101927 0.00834039  -0.291014
 -0.102219   0.988446   0.111918   0.121551
 0.0031634  -0.112183   0.993683   0.339041
         0          0          0          1
current transformation estimation
  0.994757   0.101927 0.00834039  -0.291014
 -0.102219   0.988446   0.111918   0.121551
 0.0031634  -0.112183   0.993683   0.339041
         0          0          0          1
it = 1; cloud size = 29280; selected size = 24791
JTJ =  92950.4 -42927.1 -45647.5  19505.3 -2807.42  42299.9
-42927.1  27583.8  21442.2 -9525.14   1324.4 -20755.5
-45647.5  21442.2  22517.7 -9641.93  1382.54 -20829.7
 19505.3 -9525.14 -9641.93  4282.87 -598.361  9236.44
-2807.42   1324.4  1382.54 -598.361   90.107 -1298.59
 42299.9 -20755.5 -20829.7  9236.44 -1298.59  20018.1
JTr = -138.758
 66.6306
  68.979
-33.9927
 3.96181
 -71.508
X =   0.0378539
-0.00402526
  0.0855123
   0.203131
   0.024896
 -0.0837208
transformation in this iteration
    0.996338   -0.0854987 -0.000775407     0.203131
   0.0854074     0.995619   -0.0380502     0.024896
  0.00402525    0.0378446     0.999276   -0.0837208
           0            0            0            1
current transformation estimation
   0.999851   0.0171299 -0.00202947  -0.0974733
 -0.0169317     0.99709   0.0743299    0.108159
 0.00329683  -0.0742844    0.997232    0.258503
          0           0           0           1
it = 2; cloud size = 29280; selected size = 27278
JTJ =   106650 -49865.1 -52593.7  21952.8  -3155.5  47461.4
-49865.1  32271.1  24970.3 -10820.4  1500.07 -23530.8
-52593.7  24970.3  26055.8 -10891.8  1559.69 -23452.8
 21952.8 -10820.4 -10891.8   4732.8 -661.023  10178.9
 -3155.5  1500.07  1559.69 -661.023  99.2445 -1430.34
 47461.4 -23530.8 -23452.8  10178.9 -1430.34    22006
JTr = -69.9797
 44.7118
 35.9486
-17.6539
 1.80711
-36.3435
X =   0.0231947
-0.00401701
  0.0506327
    0.13484
  0.0331909
 -0.0589205
transformation in this iteration
    0.99871  -0.0506905 -0.00283697     0.13484
  0.0506107    0.998445  -0.0233662   0.0331909
   0.004017   0.0231924    0.999723  -0.0589205
          0           0           0           1
current transformation estimation
   0.999411  -0.0332245 -0.00862379    0.031276
  0.0336208    0.998142   0.0508101    0.130209
 0.00691963  -0.0510701    0.998671    0.201628
          0           0           0           1
it = 3; cloud size = 29280; selected size = 28816
JTJ =   117818 -54770.5 -58277.6  23776.2 -3413.26  51235.2
-54770.5  34908.8  27502.5 -11604.7   1608.9 -25130.5
-58277.6  27502.5  28969.2 -11832.1  1691.92 -25385.1
 23776.2 -11604.7 -11832.1  5025.81 -703.071  10773.6
-3413.26  1608.89  1691.92 -703.071  105.135 -1516.08
 51235.2 -25130.5 -25385.1  10773.6 -1516.08  23220.2
JTr = -29.1478
 17.9882
 14.8802
  -7.199
0.732457
-14.8477
X =  0.00985068
-0.00153022
  0.0214971
  0.0555449
  0.0158326
 -0.0239884
transformation in this iteration
   0.999768  -0.0215095 -0.00131805   0.0555449
  0.0214954     0.99972 -0.00988113   0.0158326
 0.00153022  0.00985051     0.99995  -0.0239884
          0           0           0           1
current transformation estimation
  0.998446  -0.054619  -0.011031  0.0837472
 0.0550257   0.997653  0.0407425   0.144685
0.00877979 -0.0412862   0.999109    0.17896
         0          0          0          1
it = 4; cloud size = 29280; selected size = 29062
JTJ =  2.67447e+16 -7.84049e+15  -3.6278e+16 -5.63107e+15 -2.42087e+16  1.07807e+15
-7.84049e+15  1.86729e+16 -3.04592e+16  9.04303e+15 -3.79936e+15 -4.65635e+15
 -3.6278e+16 -3.04592e+16  1.52343e+17 -1.09139e+16  6.01844e+16  9.43043e+15
-5.63107e+15  9.04303e+15 -1.09139e+16  4.52285e+15  1.77921e+14 -2.18642e+15
-2.42087e+16 -3.79936e+15  6.01844e+16  1.77921e+14  2.91641e+16  1.91243e+15
 1.07807e+15 -4.65635e+15  9.43043e+15 -2.18642e+15  1.91243e+15  1.19393e+15
JTr =   1.1874e+14
-2.52785e+13
-1.84986e+14
-2.06976e+13
-1.13823e+14
 2.25993e+12
X =  0.00351562
0.000976562
 0.00117368
 0.00585938
 0.00500283
 -0.0078125
transformation in this iteration
    0.999999  -0.00117024  0.000980682   0.00585938
  0.00117368     0.999993  -0.00351447   0.00500283
-0.000976562   0.00351562     0.999993   -0.0078125
           0            0            0            1
current transformation estimation
  0.998389 -0.0558269 -0.0100988  0.0896127
 0.0561664   0.997727  0.0372179   0.149156
0.00799813 -0.0377252   0.999256   0.171573
         0          0          0          1
it = 5; cloud size = 29280; selected size = 29094
JTJ =  3.85228e+15   -8.865e+15  5.95232e+15 -4.71678e+15 -6.04553e+14  2.15227e+15
  -8.865e+15  2.04018e+16 -1.37005e+16  1.08551e+16  1.39049e+15 -4.95323e+15
 5.95232e+15 -1.37005e+16  9.20306e+15 -7.28945e+15 -9.32601e+14  3.32629e+15
-4.71678e+15  1.08551e+16 -7.28945e+15   5.7756e+15  7.39871e+14 -2.63544e+15
-6.04553e+14  1.39049e+15 -9.32601e+14  7.39871e+14  9.52667e+13 -3.37575e+14
 2.15227e+15 -4.95323e+15  3.32629e+15 -2.63544e+15 -3.37575e+14  1.20256e+15
JTr =  2.08085e+13
-4.78883e+13
 3.21585e+13
-2.54797e+13
-3.26387e+12
 1.16265e+13
X =   0.0128433
 0.00216505
-0.00500124
 0.00834517
  0.0198864
  0.0139678
transformation in this iteration
   0.999985  0.00502861  0.00210061  0.00834517
-0.00500121    0.999905  -0.0128536   0.0198864
-0.00216505   0.0128429    0.999915   0.0139678
          0           0           0           1
current transformation estimation
   0.998674  -0.0508881 -0.00781249    0.099067
  0.0510651    0.998397   0.0244209    0.166375
 0.00655723  -0.0247874    0.999671    0.187248
          0           0           0           1
it = 6; cloud size = 29280; selected size = 21480
JTJ =  83223.5 -51361.8 -36190.9  12769.4 -5665.84    37284
-51361.8  38320.7  19571.6 -5566.25  4479.49 -22739.5
-36190.9  19571.6  21818.4 -9771.92  778.722 -17248.9
 12769.4 -5566.25 -9771.92  5948.69  1186.69   7115.4
-5665.84  4479.49  778.722  1186.69  1809.97 -1772.63
   37284 -22739.5 -17248.9  7115.41 -1772.63  17702.3
JTr =  1628.63
-937.616
-809.984
 357.062
 -48.507
 769.675
X =  -0.0178255
-0.00240211
8.74875e-05
 -0.0128786
 -0.0203239
-0.00579425
transformation in this iteration
    0.999997 -4.46571e-05  -0.00240328   -0.0128786
 8.74873e-05     0.999841    0.0178244   -0.0203239
   0.0024021   -0.0178245     0.999838  -0.00579425
           0            0            0            1
current transformation estimation
  0.998653  -0.050873  -0.010216  0.0857307
 0.0512612   0.997792  0.0422348   0.149371
0.00804488 -0.0427016   0.999055   0.178696
         0          0          0          1
it = 7; cloud size = 29280; selected size = 29168
JTJ =   137245 -70218.5 -54756.4  17307.5 -9947.02  55566.1
-70218.5  57113.8  8509.89 -857.801   3813.2 -30683.9
-54756.4  8509.89  60285.3 -22262.6  6150.86 -21120.7
 17307.5 -857.801 -22262.6    10854  653.366  8318.85
-9947.02   3813.2  6150.86  653.366  4836.75  -1860.1
 55566.1 -30683.9 -21120.7  8318.85  -1860.1  24691.2
JTr = -11.5479
 43.9814
-97.1369
 29.5074
-2.42588
-1.77437
X =  0.00564612
-0.00134065
  0.0130597
  0.0252539
 -0.0116627
 -0.0125163
transformation in this iteration
   0.999914  -0.0130667 -0.00126678   0.0252539
  0.0130593    0.999899 -0.00566312  -0.0116627
 0.00134065  0.00564609    0.999983  -0.0125163
          0           0           0           1
current transformation estimation
  0.997887 -0.0638524 -0.0120326   0.108799
 0.0642522   0.997268  0.0364394     0.1378
0.00967301 -0.0371355   0.999263   0.167135
         0          0          0          1
it = 8; cloud size = 29280; selected size = 29052
JTJ =  2.84468e+16 -5.16613e+16  5.65076e+16 -2.70914e+16 -1.62882e+15  1.21495e+16
-5.16613e+16  9.41613e+16 -1.03494e+17  4.94006e+16  2.77454e+15 -2.21357e+16
 5.65076e+16 -1.03494e+17  1.14481e+17 -5.43289e+16 -2.76576e+15  2.43169e+16
-2.70914e+16  4.94006e+16 -5.43289e+16  2.59188e+16  1.44315e+15 -1.16127e+16
-1.62882e+15  2.77454e+15 -2.76576e+15  1.44315e+15  1.92091e+14 -6.57227e+14
 1.21495e+16 -2.21357e+16  2.43169e+16 -1.16127e+16 -6.57227e+14  5.20397e+15
JTr = -6.10337e+13
 1.20158e+14
 -1.4509e+14
 6.36122e+13
-1.52288e+12
-2.80187e+13
X =   0.00356821
 0.000681689
  -0.0012749
-0.000841346
   0.0530048
   0.0107272
transformation in this iteration
    0.999999   0.00127732  0.000677135 -0.000841346
  -0.0012749     0.999993  -0.00356907    0.0530048
-0.000681689    0.0035682     0.999993    0.0107272
           0            0            0            1
current transformation estimation
  0.997974 -0.0626036 -0.0113094   0.108247
  0.062945   0.997475   0.032888   0.190069
0.00922196 -0.0335333   0.999395   0.178278
         0          0          0          1
it = 9; cloud size = 29280; selected size = 29188
JTJ =  1.17564e+17 -5.70052e+16 -1.79488e+16 -3.13878e+16 -6.93931e+16  1.01122e+16
-5.70052e+16  9.88215e+16  -1.0835e+17  4.84635e+16  1.02865e+15 -2.44858e+16
-1.79488e+16  -1.0835e+17  2.32446e+17  -4.8552e+16  7.34678e+16  3.03591e+16
-3.13878e+16  4.84635e+16  -4.8552e+16  2.39647e+16   3.6365e+15 -1.18484e+16
-6.93931e+16  1.02865e+15  7.34678e+16   3.6365e+15  5.82194e+16  2.94149e+15
 1.01122e+16 -2.44858e+16  3.03591e+16 -1.18484e+16  2.94149e+15  6.26501e+15
JTr = -1.63446e+14
 1.16082e+14
-1.03395e+14
  5.7563e+13
 6.16021e+13
-2.42421e+13
X =  -0.011483
  0.011887
 0.0103372
 0.0439453
-0.0367641
  0.119141
transformation in this iteration
  0.999876 -0.0104728  0.0117666  0.0439453
 0.0103363   0.999879   0.011605 -0.0367641
-0.0118867  -0.011482   0.999863   0.119141
         0          0          0          1
current transformation estimation
     0.9973  -0.0734368 0.000106994    0.152286
  0.0733597    0.996318   0.0443652     0.15647
-0.00336464  -0.0442375    0.999015    0.293926
          0           0           0           1
it = 10; cloud size = 29280; selected size = 0
JTJ = 0 0 0 0 0 0
0 0 0 0 0 0
0 0 0 0 0 0
0 0 0 0 0 0
0 0 0 0 0 0
0 0 0 0 0 0
JTr = 0
0
0
0
0
0
X = 0
0
0
0
0
0
transformation in this iteration
1 0 0 0
0 1 0 0
0 0 1 0
0 0 0 1
current transformation estimation
     0.9973  -0.0734368 0.000106994    0.152286
  0.0733597    0.996318   0.0443652     0.15647
-0.00336464  -0.0442375    0.999015    0.293926
          0           0           0           1
```


# Ceres

## Introduction

* Ceres provides good official tutorials: <http://ceres-solver.org/nnls_tutorial.html>
* Example project that uses Ceres: <https://github.com/hanzheteng/A-LOAM>


# g2o

## Introduction

* <https://github.com/RainerKuemmerle/g2o>


# GTSAM

## Introduction

* Official tutorials: <https://gtsam.org/tutorials/intro.html>
* Example project that uses GTSAM: <https://github.com/hanzheteng/LeGO-LOAM>

## Examples

### Speed up Key operations by [TBB](https://en.wikipedia.org/wiki/Threading_Building_Blocks) library

```cpp
// inference/Factor.h 
class Factor{
  protected:
    /// The keys involved in this factor
    KeyVector keys_;
}

// inference/Key.h
typedef FastVector<Key> KeyVector;

// base/types.h
typedef std::uint64_t Key;         /// Integer nonlinear key type
typedef std::uint64_t FactorIndex; /// Integer nonlinear factor index type
typedef ptrdiff_t DenseIndex;      /// The index type for Eigen objects

// base/FastVector.h
// FastVector is a type alias to a std::vector with a custom memory allocator.
// The particular allocator depends on GTSAM's cmake configuration.
template <typename T>
using FastVector =
    std::vector<T, typename internal::FastDefaultVectorAllocator<T>::type>;

// base/FastDefaultAllocator.h
/// Default allocator for vector types (we never use boost pool for vectors)
    template<typename T>
    struct FastDefaultVectorAllocator
    {
#if defined GTSAM_ALLOCATOR_TBB
      typedef tbb::tbb_allocator<T> type;
      static const bool isBoost = false;
      static const bool isTBB = true;
      static const bool isSTL = false;
#else
      typedef std::allocator<T> type;
      static const bool isBoost = false;
      static const bool isTBB = false;
      static const bool isSTL = true;
#endif
    };
```


# Website

Taking some notes regarding building a website.

### Domain

First you may need to buy and host a domain. For example, hanzheteng.com

* To check domain status, you can use [whois](https://www.whois.com/). This can provide you with some information, but not a service to host your own domain.
* To register or buy a new domain, you can use [GoDaddy](https://www.godaddy.com/) or [Aliyun](https://wanwang.aliyun.com/). They are domain service providers, and can help you host and manage this domain (including setting up DNS server).

### Jekyll Local Setup

For the latest version (22.04) of Ubuntu system, you should be able to set up everything by following this official latest tutorial: [Quick Start](https://jekyllrb.com/docs/), [Ubuntu Installation](https://jekyllrb.com/docs/installation/ubuntu/)

For Ubuntu 20, you may encountered some errors in `gem install jekyll bundler`. To address this, just to install a previous version of jekyll. Either of the following way shall work.

```
gem install jekyll -v 3.9.3
gem install jekyll --version="~> 4.2.0"
```

Then just follow the rest of [Quick Start ](https://jekyllrb.com/docs/)to compile and run the web pages locally.

### Github Pages

There are some issues with the rendering of Github Pages on the Github remote server. The webpages can build and render correctly in local environment, but not on the Internet after pushing to Github repo. Two things to change:

* In markdown source files (e.g., `index.md`), change the keyword `layout` to `layouts`. This is may be a legacy issue. Reference: [Stackoverflow question](https://stackoverflow.com/questions/39193199/build-warning-layout-requested-does-not-exist).
* In the `_config.yml` file, `baseurl` and `url` have to be set propely to a remote address. For example:&#x20;

```
baseurl: "/"
url: "http://ucr-robotics.github.io"
```

### Resources

* Website builder: [WordPress](https://wordpress.com/), [Google Sites](https://sites.google.com/)
* Website builder for hackers: [Github pages](https://pages.github.com/), [Jekyll](https://jekyllrb.com/), [Hugo](https://gohugo.io/), [Hexo](https://hexo.io/docs/index.html)
* HTML templates: [TEMPLATED](https://templated.co/)
* Jekyll templates: [Jekyll themes](https://jekyllrb.com/resources/#themes)
* Hugo templates:  [Congo](https://jpanther.github.io/congo/), [Wowchemy](https://wowchemy.com/templates/)


# SLAM


# K-D Tree

### Binary Search Tree (BST)

#### Tree

* In graph theory, a **tree** is a connected **acyclic undirected graph**, and a **forest** is a disjoint union of trees.
* In computer science, a **tree** is a widely used abstract data type (ADT) that simulates a hierarchical tree structure, with a root value and subtrees of children with a parent node, represented as a set of linked nodes. (This "linked" relation can be implemented by a linked list or an array with certain indexing rules.)

#### Binary Tree

* A binary tree is a tree data structure in which each node has at most two children. (This is the only constraint.)

**Binary Search Tree (BST)**

* In computer science, a binary search tree (BST), also called an **ordered** or **sorted** binary tree, is a rooted binary tree data structure whose internal nodes each store a key greater than all the keys in the node's left subtree and less than those in its right subtree.&#x20;
* Used for sorting: the construction step takes O(n log n) time, then the sorted list can be obtained by traversing all nodes in O(log n) time.&#x20;
* | Algorithm |   | Average  | Worst case |
  | --------- | - | -------- | ---------- |
  | Space     |   | O(n)     | O(n)       |
  | Search    |   | O(log n) | O(n)       |
  | Insert    |   | O(log n) | O(n)       |
  | Delete    |   | O(log n) | O(n)       |

### K-D Tree

* In computer science, a k-d tree (short for k-dimensional tree) is a space-partitioning data structure for organizing points in a k-dimensional space
* k-d trees are a special case of binary space partitioning trees.
* | Algorithm |   | Average  | Worst case |
  | --------- | - | -------- | ---------- |
  | Space     |   | O(n)     | O(n)       |
  | Search    |   | O(log n) | O(n)       |
  | Insert    |   | O(log n) | O(n)       |
  | Delete    |   | O(log n) | O(n)       |

### Comparison of various data structures

![](/files/-Mglybe2N54822Dvyf1p)

### Discussions on K-D Tree

* It is said that Euclidean distance is a bad metric in high dimensional space.
* There are debates on using K-D Tree for high dimensional data.
  * Against: [When Is "Nearest Neighbor" Meaningful?](https://link.springer.com/content/pdf/10.1007%2F3-540-49257-7_15.pdf) by Kevin Beyer et al.
  * For: [When is ‘nearest neighbour’ meaningful: A converse theorem and implications](https://www.sciencedirect.com/science/article/pii/S0885064X09000260) by Robert Durrant et al. ([Bob's answer on Stack Exchange](https://stats.stackexchange.com/a/10976))
* More discussions on [Stackoverflow: Nearest neighbors in high-dimensional data?](https://stackoverflow.com/questions/5751114/nearest-neighbors-in-high-dimensional-data)


# Octree

K-D Tree may be good for high-dimensional space; however, for 3D point clouds, Octree may provide better performance in some cases. In the following, we report some benchmark results on LiDAR point clouds.&#x20;

### Experiment 1: Ouster LiDAR cloud

* Target cloud size = 65536, source cloud size = 3000.
* For each point in the source cloud, find its neighbor point in the target cloud.
* The scale is roughly 10-50 meters, in a structured environment.
* 13-gen Intel evo i7 CPU (dell xps laptop), single threading.
* Both data structures are implemented in PCL.

<table><thead><tr><th width="278.3333333333333">Operation</th><th>K-D Tree</th><th>Octree (leaf size 0.01 m)</th><th>Octree (leaf size 0.001m)</th></tr></thead><tbody><tr><td>Build Tree</td><td>7 ms</td><td>8 ms</td><td>12 ms</td></tr><tr><td>Approximate NN Search (k=1)</td><td>(not support)</td><td>94 ms</td><td>96 ms</td></tr><tr><td>K-NN Search (k=1)</td><td>124 ms</td><td>390 ms</td><td>408 ms</td></tr><tr><td>K-NN Search (k=10)</td><td>124 ms</td><td>395 ms</td><td>452 ms</td></tr><tr><td>K-NN Search (k=100)</td><td>134 ms</td><td>573 ms</td><td>698 ms</td></tr><tr><td>Radius Search (r=0.01 m)</td><td>1789 ms</td><td>126 ms</td><td>126 ms</td></tr><tr><td>Radius Search (r=0.1 m)</td><td>1799 ms</td><td>129 ms</td><td>130 ms</td></tr><tr><td>Radius Search (r=1 m)</td><td>2011 ms</td><td>415 ms</td><td>861 ms</td></tr></tbody></table>

Other observations from ablation study:

* Time consumption of tree building is linear in target cloud size.
* Time consumption of each search method is also almost linear in source/query cloud size. (i.e. size = 30000 can roughly take 10x more time than size = 3000).
* The location and shape of source query cloud does not affect much on running time.
* PCL Point Type (PointXYZ vs. PointXYZRGBNormal) does not affect much on running time.

Other notes:

* This result can be accelerated by multi-threading, e.g., OpenMP.
* The `approxNearestSearch` method in Octree can only find the nearest point (k=1 case), and uses tree nodes as the approximate result similar to the radius search to speed up.

### Experiment 2: Velodyne LiDAR Cloud (VLP-16)

* Target cloud size = 28209 (NaN points removed), source cloud size = 3000.
* For each point in the source cloud, find its neighbor point in the target cloud.
* The scale is roughly 10-50 meters, in a structured environment.
* 13-gen Intel evo i7 CPU (dell xps laptop), single threading.
* Both data structures are implemented in PCL.

<table><thead><tr><th width="278.3333333333333">Operation</th><th>K-D Tree</th><th>Octree (leaf size 0.01 m)</th><th>Octree (leaf size 0.001m)</th></tr></thead><tbody><tr><td>Build Tree</td><td>4 ms</td><td>6 ms</td><td>9 ms</td></tr><tr><td>Approximate NN Search (k=1)</td><td>(not support)</td><td>1.5 ms</td><td>1.5 ms</td></tr><tr><td>K-NN Search (k=1)</td><td>3 ms</td><td>23 ms</td><td>23 ms</td></tr><tr><td>K-NN Search (k=10)</td><td>5 ms</td><td>41 ms</td><td>57 ms</td></tr><tr><td>K-NN Search (k=100)</td><td>21 ms</td><td>407 ms</td><td>539 ms</td></tr><tr><td>Radius Search (r=0.01 m)</td><td>1 ms</td><td>2 ms</td><td>2 ms</td></tr><tr><td>Radius Search (r=0.1 m)</td><td>2 ms</td><td>4 ms</td><td>5 ms</td></tr><tr><td>Radius Search (r=1 m)</td><td>34 ms</td><td>59 ms</td><td>138 ms</td></tr></tbody></table>

Notes (based on new observations):

* If using method K-NN Search (k=1), Octree and K-D Tree can get almost the same neighbor points as the output result.
* If using method Approximate NN Search (k=1) from Octree, the result will be approximated and most likely different from the K-NN Search (k=1) result. This method is OK to be used in finding the neighbor points (to constitute a submap) from a global map, but it should not be used internally in an ICP algorithm in each iteration, as it can make the algorithm not to converge.

### Benchmark Code

```cpp
#include <iostream>
#include <chrono>
#include <iomanip>
#include <pcl/io/pcd_io.h>
#include <pcl/io/ply_io.h>
#include <pcl/point_cloud.h>
#include <pcl/octree/octree_search.h>
#include <pcl/kdtree/kdtree_flann.h>

using PointT = pcl::PointXYZRGBNormal;
using CloudT = pcl::PointCloud<PointT>;
using CloudTPtr = pcl::PointCloud<PointT>::Ptr;

void LoadPointCloud(const std::string& filename, CloudTPtr cloud) {
  if (filename.substr(filename.find_last_of(".") + 1) == "pcd") {
    pcl::io::loadPCDFile(filename, *cloud);
  } else {
    pcl::io::loadPLYFile(filename, *cloud);
  }
}

template<typename SearchTree>
double TimeTakenForSearch(SearchTree& tree, CloudTPtr source, int k) {
  std::vector<int> pointIdxNKNSearch;
  std::vector<float> pointNKNSquaredDistance;

  auto start_time = std::chrono::high_resolution_clock::now();
  for (const auto& pt : source->points) {
    tree.nearestKSearch(pt, k, pointIdxNKNSearch, pointNKNSquaredDistance);
  }
  auto end_time = std::chrono::high_resolution_clock::now();
  return std::chrono::duration<double, std::micro>(end_time - start_time).count() / 1000.0;
}

template<typename SearchTree>
double TimeTakenForRadiusSearch(SearchTree& tree, CloudTPtr source, float radius) {
  std::vector<int> pointIdxRadiusSearch;
  std::vector<float> pointRadiusSquaredDistance;

  auto start_time = std::chrono::high_resolution_clock::now();
  for (const auto& pt : source->points) {
    tree.radiusSearch(pt, radius, pointIdxRadiusSearch, pointRadiusSquaredDistance);
  }
  auto end_time = std::chrono::high_resolution_clock::now();
  return std::chrono::duration<double, std::micro>(end_time - start_time).count() / 1000.0;
}

int main(int argc, char** argv) {
  if (argc != 3) {
    std::cerr << "Usage: " << argv[0] << " <source.pcd/ply> <target.pcd/ply>" << std::endl;
    return -1;
  }

  CloudTPtr source_original(new CloudT);
  CloudTPtr target(new CloudT);
  LoadPointCloud(argv[1], source_original);
  LoadPointCloud(argv[2], target);

  std::cout << "Source cloud size = " << source_original->size() << std::endl;
  std::cout << "Target cloud size = " << target->size() << std::endl;

  // Reduce source cloud size
  CloudTPtr source(new CloudT);
  int limit = std::min(3000, static_cast<int>(source_original->size()));
  for (int i = 0; i < limit; i++) {
    source->points.push_back(source_original->points[i]);
  }
  source->width = source->points.size();
  source->height = 1;
  source->is_dense = true;
  std::cout << "Reduced source cloud size = " << source->points.size() << std::endl;

  // Octree benchmark
  pcl::octree::OctreePointCloudSearch<PointT> octree(0.01f);
  auto start_time = std::chrono::high_resolution_clock::now();
  octree.setInputCloud(target);
  octree.addPointsFromInputCloud();
  auto end_time = std::chrono::high_resolution_clock::now();
  auto duration = std::chrono::duration<double, std::micro>(end_time - start_time).count() / 1000.0;
  std::cout << std::fixed << std::setprecision(3) << "Octree build took: " << duration << "ms" << std::endl;

  // Approximate nearest neighbor search using octree for entire source cloud
  start_time = std::chrono::high_resolution_clock::now();
  int result_index;
  float sqr_distance;
  for (const auto& pt : source->points) {
    octree.approxNearestSearch(pt, result_index, sqr_distance);
  }
  end_time = std::chrono::high_resolution_clock::now();
  duration = std::chrono::duration<double, std::micro>(end_time - start_time).count() / 1000.0;
  std::cout << "Octree approximate nearest neighbor search for entire source took: " << duration << "ms" << std::endl;

  std::cout << "Octree k=1 search for entire source took: " << TimeTakenForSearch(octree, source, 1) << "ms" << std::endl;
  std::cout << "Octree k=10 search for entire source took: " << TimeTakenForSearch(octree, source, 10) << "ms" << std::endl;
  std::cout << "Octree k=100 search for entire source took: " << TimeTakenForSearch(octree, source, 100) << "ms" << std::endl;
  std::cout << "Octree radius search 0.01 for entire source took: " << TimeTakenForRadiusSearch(octree, source, 0.01f) << "ms" << std::endl;
  std::cout << "Octree radius search 0.1 for entire source took: " << TimeTakenForRadiusSearch(octree, source, 0.1f) << "ms" << std::endl;
  std::cout << "Octree radius search 1.0 for entire source took: " << TimeTakenForRadiusSearch(octree, source, 1.0f) << "ms" << std::endl;

  // KdTree benchmark
  pcl::KdTreeFLANN<PointT> kdtree;
  start_time = std::chrono::high_resolution_clock::now();
  kdtree.setInputCloud(target);
  end_time = std::chrono::high_resolution_clock::now();
  duration = std::chrono::duration<double, std::micro>(end_time - start_time).count() / 1000.0;
  std::cout << "KdTree build took: " << duration << "ms" << std::endl;

  std::cout << "KdTree k=1 search for entire source took: " << TimeTakenForSearch(kdtree, source, 1) << "ms" << std::endl;
  std::cout << "KdTree k=10 search for entire source took: " << TimeTakenForSearch(kdtree, source, 10) << "ms" << std::endl;
  std::cout << "KdTree k=100 search for entire source took: " << TimeTakenForSearch(kdtree, source, 100) << "ms" << std::endl;
  std::cout << "KdTree radius search 0.01 for entire source took: " << TimeTakenForRadiusSearch(kdtree, source, 0.01f) << "ms" << std::endl;
  std::cout << "KdTree radius search 0.1 for entire source took: " << TimeTakenForRadiusSearch(kdtree, source, 0.1f) << "ms" << std::endl;
  std::cout << "KdTree radius search 1.0 for entire source took: " << TimeTakenForRadiusSearch(kdtree, source, 1.0f) << "ms" << std::endl;

  return 0;
}
```


# Bag of Words

## DBoW Library

The library is composed of two main classes: `Vocabulary` and `Database`. The former is trained offline with numerous images, whereas the latter can be established/expanded online. Both structures can be saved in binary or text format.

#### Weighting

Words in the vocabulary and in bag-of-words vectors are weighted. There are four weighting measures implemented to set a word weight *wi*:

* Term frequency (*tf*): [![w\_i = \frac{n\_{id}}{n\_d}](https://raw.githubusercontent.com/dorian3d/dorian3d.github.io/master/other/images/tf.gif)](https://raw.githubusercontent.com/dorian3d/dorian3d.github.io/master/other/images/tf.gif), [![n\_{id}](https://raw.githubusercontent.com/dorian3d/dorian3d.github.io/master/other/images/nid.gif)](https://raw.githubusercontent.com/dorian3d/dorian3d.github.io/master/other/images/nid.gif): number of occurrences of word *i* in document *d*, [![n\_d](https://raw.githubusercontent.com/dorian3d/dorian3d.github.io/master/other/images/nd.gif)](https://raw.githubusercontent.com/dorian3d/dorian3d.github.io/master/other/images/nd.gif): number of words in document *d*.
* Inverse document frequency (*idf*): [![w\_i = log(\frac{N}{N\_i})](https://raw.githubusercontent.com/dorian3d/dorian3d.github.io/master/other/images/idf.gif)](https://raw.githubusercontent.com/dorian3d/dorian3d.github.io/master/other/images/idf.gif), [![N](https://raw.githubusercontent.com/dorian3d/dorian3d.github.io/master/other/images/N.gif)](https://raw.githubusercontent.com/dorian3d/dorian3d.github.io/master/other/images/N.gif): number of documents, [![N\_i](https://raw.githubusercontent.com/dorian3d/dorian3d.github.io/master/other/images/Ni.gif)](https://raw.githubusercontent.com/dorian3d/dorian3d.github.io/master/other/images/Ni.gif): number of documents containing word *i*.
* Term frequency -- inverse document frequency (*tf-idf*): [![w\_i = \frac{n\_{id}}{n\_d} log(\frac{N}{N\_i}](https://raw.githubusercontent.com/dorian3d/dorian3d.github.io/master/other/images/tf-idf.gif)](https://raw.githubusercontent.com/dorian3d/dorian3d.github.io/master/other/images/tf-idf.gif).
* Binary: [![w\_i = 1 if word i is present; 0 otherwise](https://raw.githubusercontent.com/dorian3d/dorian3d.github.io/master/other/images/binary.gif)](https://raw.githubusercontent.com/dorian3d/dorian3d.github.io/master/other/images/binary.gif)

DBow calculates *N* and *Ni* according to the number of images provided when the vocabulary is created. These values are not changed and are independent of how many entries a `Database` object contains.

#### Scoring&#x20;

A score is calculated when two vectors are compared by means of a `Vocabulary` or when a `Database` is queried. These are the metrics implemented to calculate the score *s* between two vectors *v* and *w* (from now on, *v\** and *w\** denote vectors normalized with the L1-norm):

* Dot product: [![Dot product](https://raw.githubusercontent.com/dorian3d/dorian3d.github.io/master/other/images/dot.gif)](https://raw.githubusercontent.com/dorian3d/dorian3d.github.io/master/other/images/dot.gif)
* L1-norm: [![L1-norm](https://raw.githubusercontent.com/dorian3d/dorian3d.github.io/master/other/images/L1.gif)](https://raw.githubusercontent.com/dorian3d/dorian3d.github.io/master/other/images/L1.gif)
* L2-norm: [![L2-norm](https://raw.githubusercontent.com/dorian3d/dorian3d.github.io/master/other/images/L2.gif)](https://raw.githubusercontent.com/dorian3d/dorian3d.github.io/master/other/images/L2.gif)
* Bhattacharyya coefficient: [![Bhattacharyya coefficient](https://raw.githubusercontent.com/dorian3d/dorian3d.github.io/master/other/images/bhat.gif)](https://raw.githubusercontent.com/dorian3d/dorian3d.github.io/master/other/images/bhat.gif)
* χ² (chi-square) distance: [![Chi square distance](https://raw.githubusercontent.com/dorian3d/dorian3d.github.io/master/other/images/chisq_extended.gif)](https://raw.githubusercontent.com/dorian3d/dorian3d.github.io/master/other/images/chisq_extended.gif)
* KL-divergence: [![KL-divergence](https://raw.githubusercontent.com/dorian3d/dorian3d.github.io/master/other/images/kl_extended.gif)](https://raw.githubusercontent.com/dorian3d/dorian3d.github.io/master/other/images/kl_extended.gif)

The default configuration when creating a vocabulary is *tf-idf*, L1-norm.

### References

* [dorian3d/DBow](https://github.com/dorian3d/DBow)
* [dorian3d/DBoW2](https://github.com/dorian3d/DBoW2)
* [rmsalinas/DBow3](https://github.com/rmsalinas/DBow3)
* [rmsalinas/fbow](https://github.com/rmsalinas/fbow)


# Distance Measures

## 9 Common Measures in Data Science

![](/files/FZq6CQtbocRNyu5T5pSn)

Reference: [9 Distance Measures in Data Science](https://towardsdatascience.com/9-distance-measures-in-data-science-918109d069fa)

## Rotation: Geodesic Distance

$$
\| \rm{arccos} (\frac{ tr(\hat{R}^T R^o)-1}{2} )|
$$

* Geodesic distance on the unit sphere: SO(3) is a compact Lie group and it has a natural Riemannian metric; that is, an inner product on its tangent space so(3) at every point. \[Huynh 2009]&#x20;
* This metric tries to find$$R$$such that $$R\_1 = RR\_2$$*.* Therefore, $$R = R\_1R\_2^{-1} = R\_1R\_2^T$$.
* In the second step, it is the matrix logorithm from SO(3) to so(3). theta = arccos(1/2(tr(R)-1)). See the rule (c) before equation (3.61) in the Modern Robotics textbook.&#x20;
* Lastly, the range of the arccos function is \[0, pi], so there is no need to add abs().

## Feature Distance

* Humming Distance for binary descriptors
* L1 or L2 Distance for histogram based descriptors (a float vector)
* Sum of Squared Distance (SSD) for image patches (in fact it is still the L2 distance)
  * can give good scores to very ambiguous (bad) matches
* Improvement: use the ratio distance = SSD(f1, f2) / SSD(f1, f2’)
  * gives small values for ambiguous matches

#### Normalized Euclidean Distance vs. Normalized Cross Correlation

The normalized Euclidean distance is the distance between two normalized vectors that have been normalized to length one. If the vectors are identical then the distance is `0`, if the vectors point in opposite directions the distance is `2`, and if the vectors are orthogonal (perpendicular) the distance is `sqrt(2)`. It is a positive definite scalar value between `0` and `2`.

[![NED equation](https://i.stack.imgur.com/QSIBn.png)](https://i.stack.imgur.com/QSIBn.png)

The normalized cross-correlation is the dot product between the two normalized vectors. If the vectors are identical, then the correlation is `1`, if the vectors point in opposite directions the correlation is `-1`, and if the vectors are orthogonal (perpendicular) the correlation is `0`. It is a scalar value between `-1` and `1`. This is also called the **cosine similarity**.&#x20;

[![NCC equation](https://i.stack.imgur.com/XHjSu.png)](https://i.stack.imgur.com/XHjSu.png)

References: <https://datascience.stackexchange.com/a/6545>

## Probability Distribution

* Probability Mass Functions (PMF) are used to describe **discrete** probability distributions.&#x20;
* Probability Density Functions (PDF) are used to describe **continuous** probability distributions.

#### KL-Divergence (Relative Entropy)

* The relative entropy or Kullback-Leibler (KL) distance between two PMFs p(x) and q(x) (that are defined on the same alphabet) is:&#x20;

![](/files/nEGmtb4VRmJB35c6M4ru)

* KL-distance is not a rigorous distance metric, because it does not meet all of the properties (e.g., does not satisfy the triangular inequality and it is not symmetric), but it is (to some degree) a distance measure because it does meet some of the properties (e.g., always non-negative).
* This is not the only measure between two distributions, but it is the most important one in information theory.
* References: EE250 Information Theory Lecture Notes

#### Earth Mover's Distance (Wasserstein metric)

* In statistics, the earth mover's distance (EMD) is a measure of the distance between two probability distributions over a region D. In mathematics, this is known as the Wasserstein metric.&#x20;
* Informally, if the distributions are interpreted as two different ways of piling up a certain amount of earth (dirt) over the region D, the EMD is the minimum cost of turning one pile into the other; where the cost is assumed to be the amount of dirt moved times the distance by which it is moved.
* The above definition is valid only if the two distributions have the same integral (informally, if the two piles have the same amount of dirt), as in **normalized histograms or probability density functions**. In that case, the EMD is equivalent to the 1st Mallows distance or 1st Wasserstein distance between the two distributions.
* Related Keywords: Transportation Theory, Optimal Transport
* References: <https://en.wikipedia.org/wiki/Transportation_theory_(mathematics)>; <https://www.math.cmu.edu/~mthorpe/OTNotes>
* It is said that one can possibly replace KL-Divergence in machine learning papers with this new metric and attain better performance.


# Coordinate Systems

### ENU

* a local projection from sphere to cylinder (base point is not fixed)
* z axis is always the opposite direction of gravity (z can represent altitude)
* the far from base point, the greater distortion of the space

### GENU

* a global projection from sphere to cylinder (base point is fixed)
* the far from base point, the greater difference between z axis and gravity (z cannot represent altitude anymore)
* no distortion of the space, no matter how far from the base point

### ECEF

* the earth-centered coordinate system
* no distortion, but not intuitive for objects moving on earth surface

### References

* <https://zhuanlan.zhihu.com/p/33295151>
* <https://en.wikipedia.org/wiki/Mercator_projection>
* <https://en.wikipedia.org/wiki/Local_tangent_plane_coordinates>
* <https://en.wikipedia.org/wiki/Geographic_coordinate_conversion#From_ECEF_to_ENU>

![](/files/2hWFwwo2DM1ibaaZbcWx)

![](/files/fAYaXPbZNY5OtcROuBP4)


# LOAM

## LOAM (A-LOAM)

Note: The following introduces only LiDAR-related algorithms, and omits the IMU-related component in the original LOAM algorithm.

The algorithm consists of three parts: Scan Registration (pre-processing, feature extraction), Odometry (high speed front end), and Mapping (low speed back end). The input sensing data flows through three components one by one via ROS topics.

### Scan Registration

* For each new scan received, transform from ROS msg into PCL point cloud, remove outlier points, compute and save metadata, rectify the point cloud if possible.
* For each point in the point cloud, compute its curvature as follows.

```cpp
float diffX = laserCloud->points[i - 5].x + laserCloud->points[i - 4].x
            + laserCloud->points[i - 3].x + laserCloud->points[i - 2].x
            + laserCloud->points[i - 1].x - 10 * laserCloud->points[i].x
            + laserCloud->points[i + 1].x + laserCloud->points[i + 2].x
            + laserCloud->points[i + 3].x + laserCloud->points[i + 4].x
            + laserCloud->points[i + 5].x;
float diffY = ...;  float diffZ = ...;
cloudCurvature[i] = diffX * diffX + diffY * diffY + diffZ * diffZ;
```

* For each scan line/beam, divide the point cloud into six sectors (each takes 1/6 of start-end index diff), and extract sharp (corner) and flat (plane/surface) features as follows.
  * In each sector, sort cloud index according to curvature. The top 2 largest curvature points (if points are not selected and curvature > 0.1) are marked as **sharp**, the top 20 largest curvature points are marked as **less sharp** (including the top 2 sharp points), the top 4 smallest curvature points (with additional condition curvature < 0.1) are marked as **flat**, and all the rest points plus the top 4 flat points will be down sampled and marked as **less flat**.
  * After each feature extraction, there will be a non-maximum suppression step to mark neighbor 10 points (+-5) to be already selected (marking will stop at points that are 0.05 squared distance away from currently picked point), so that they will not be picked in the next iteration for feature extraction.
  * For example for VLP-16 LiDAR, there could be 384 flat feature points and 192 sharp feature points extracted in a point cloud frame (if all meet the 0.1 curvature threshold).
  * A downsample operation (of leaf size 0.2m) is applied to each scan of **less flat** points, and then all downsampled scans are combined together (into one point cloud) to be published to odometry.

### Odometry

* We use TransformToStart and TransformToEnd to undistort points, such that they can be treated as being captured at the beginning/end of a scan. (TransformToStart is useful in the second round of optimization. In the first round of optimization, TransformToStart is using the estimation from last frame, being applied in this frame assuming an uniform motion.)
* Synchronize the ROS topics received for sharp, less sharp, flat, less flat points. (Pick the one that shares the timestamp with others.) Run Ceres solver **twice** to quickly compute a rough estimate of the current pose (10Hz).
  * For each **sharp** point in the current frame, run TransformToStart and find two closest (3D distance) points in the previous **less sharp** frame as a line correspondence. One is found by Kd-tree search, and the other is found by line search in nearby scans, based on the distance to this kd-tree found one. Two points must be found in two different lines/scans. Then compute point-to-line distance and add this residual to Ceres solver. See LOAM paper Fig. 7(a) for details.
  * For each **flat** point in the current frame, run TransformToStart and find three closest (3D distance) points in the previous **less flat** frame as a plane correspondence. One is found by Kd-tree search, and the other two is found by line search. Two points should be on the same line/scan, and the third one should be on a different (but neighbor) line/scan. Then we compute point-to-plane distance and add this residual to Ceres solver. See LOAM paper Fig. 7(b) for details.&#x20;
  * Solve the transformation by Ceres, update feature points stored in the last frame, and publish to ROS topics.

### Mapping

* Point cloud map is maintained manually using array of pointers (of type pcl::PointCloud).
  * Since the array index must always be positive, we shift the center of the map by an offset in all x, y, z axes.&#x20;
  * In each iteration, if the current map boundary is about to reach the maximum map boundary, we shift the whole map to make it away. This shifting is in fact done in a smart way that we only shift the pointers to the point cloud, not the actual points.&#x20;
  * The entire map is divided into cubes of 50m, and the maximum map boundary is set to allow for 21 cubes in x, y axes and 11 cubes in z axis. (1050 meter horizontal square.)
* Synchronize incoming ROS msgs, adjust map and enable only nearby feature points on the map, downsample them and run Ceres solver **twice**.&#x20;
  * High-level idea: Perform SVD for each feature point. For line features, one eigenvalue should be significantly greater than the other two. For plane features, there should be two large eigenvalues with the third one significantly smaller. Compute residual for line and plane features, stack into Ceres solver and solve them together in the end. Update map corner/surface clouds using estimated transformation.
  * Details: Receive **less sharp** points (of the last frame) from odometry, and downsample them with a leaf size of 0.2m. For each point `p` in this cloud, run PointAssociateToMap to obtain `pm` . Find 5 points closest to `pm` from map corner cloud. Compute the center of 5 points, and estimate the line/edge direction by picking the largest eigenvalue of the covariance matrix of the 5 points. Compute two artificial points by adding +- 0.1 offset to the center point and feed them together with the original point `p` to Ceres functor.
  * Details: Receive **less flat** points (of the last frame) from odometry, and downsample them with a leaf size of 0.4m. For each point `p` in this cloud, run PointAssociateToMap to obtain `pm` . Find 5 points closest to `pm` from map corner cloud and organize them into a 5-by-3 matrix `M`. Find the normal vector `n` by solving a system of linear equations `M * n = -1`. Finally, feed the original point `p`, normalized normal vector, and the reciprocal of the norm of the (non-unit) normal vector to Ceres functor. (See below plane fitting algorithms for details.)
  * Two outlier checks apply here. 1) Skip point `p` if the squared distance between point `pm` and its 5th neighbor is greater than one. 2) Skip point `p` if the line/plane correspondence is not healthy. For line/edge features, the largest eigenvalue should be at least 3 times greater than the second largest eigenvalue. For plane features, the dot product between any point fitted to the plane (5 neighbor points of `pm`) and the (non-unit) normal vector, should be greater than 0.2 (an experimental threshold) minus the reciprocal of the norm of the (non-unit) normal vector.

### Math

The overall cost function consists of two parts: sum of all point-to-line distances and sum of all point-to-plane distances. Mathematically,&#x20;

$$
f(\mathbf{x})=\sum\_i d\_\mathcal{E}(p\_i(\mathbf{x})) + \sum\_j d\_\mathcal{H}(p\_j(\mathbf{x}))
$$

#### Point-to-line Distance

The area of a parallelogram can be computed by two ways: 1) the cross product of OA and OB (and then take the norm), and 2) AB multiplied by OD. Therefore, we can compute OD by |OA x OB|/|AB|. This is the approached adopted in the algorithm, where point O is available in the current frame, and two points A and B are the correspondence in the previous frame.

#### Point-to-plane Distance

If we have access to the normal vector of a plane, then the point-to-plane distance can be readily computed by the dot product with the normal vector. Specifically, O is the point out of the plane, and another point in the plane is known (for example A). Then the distance between O and the plane is the dot product between AO and the (normalized/unit) normal vector of the plane.

In the Odometry algorithm, where we have access to three points on the plane, the normal vector of the plane can be computed by a cross product (i.e. for any three points A, B, and C, just do ABxAC). In the Mapping algorithm, where we have access to more then three (e.g., five) points on the plane, the normal vector can be computed by SVD.

![](/files/WYOSkxeaar1uOwicUTSd)

#### Plane Fitting Algorithms

Recall that the equation for a plane passing through origin is Ax + By + Cz = 0, where (x, y, z) can be any point on the plane and (A, B, C) is the normal vector perpendicular to this plane.

The equation for a general plane (that may or may not pass through origin) is Ax + By + Cz + D = 0, where the additional coefficient D represents how far the plane is away from the origin, along the direction of the normal vector of the plane. *\[Note that in this equation (A, B, C) forms a unit normal vector.]*

Now, we can apply a trick here and fit the plane using only provided point coordinates. Divide both sides by D and rearrange this term to the right-hand side. This leads to A/D x + B/D y + C/D z = -1. *\[Note that in this equation (A/D, B/D, C/D) forms a normal vector with length 1/D.]*

We can set up a system of linear equations accordingly, and then solve it by an Eigen solver in C++ as follows.

```cpp
// Example for 5 points
Eigen::Matrix<double, 5, 3> matA; // row: 5 points; column: xyz coordinates
Eigen::Matrix<double, 5, 1> matB = -1 * Eigen::Matrix<double, 5, 1>::Ones();

// Find the plane normal
Eigen::Vector3d normal = matA.colPivHouseholderQr().solve(matB);

// Check if the fitting is healthy
double D = 1 / normal.norm();
normal.normalize(); // normal is a unit vector from now on
bool planeValid = true;
for (int i = 0; i < 5; ++i) { // compare Ax + By + Cz + D with 0.2 (ideally Ax + By + Cz + D = 0)
  if ( fabs( normal(0)*matA(i, 0) + normal(1)*matA(i, 1) + normal(2)*matA(i, 2) + D) > 0.2) {
    planeValid = false; // 0.2 is an experimental threshold; can be tuned
    break;
  }
}
```

References: [Wikipedia: Distance\_from\_a\_point\_to\_a\_line](https://en.wikipedia.org/wiki/Distance_from_a_point_to_a_line#Another_vector_formulation); [Distances to lines and planes](https://web.ma.utexas.edu/users/m408m/Display12-5-5.shtml); [Finding the normal to a plane (cross product for three points)](https://web.ma.utexas.edu/users/m408m/Display12-5-4.shtml); [Lesson Explainer: Equation of a Plane: Vector, Scalar, and General Forms](https://www.nagwa.com/en/explainers/373101390857/); [Stackoverflow: 3D Least Squares Plane](https://stackoverflow.com/a/72368368/13980439); [Stackoverflow: How to fit a plane to a 3D point cloud?](https://stackoverflow.com/a/72358807/13980439);&#x20;

### Code Organization

* In LOAM, they use float array `transform[6]` and `transformSum[6]` for local and global transformation respectively. Optimization is solved by their code.&#x20;
* In A-LOAM, they use `Eigen::Quaterniond q_w_curr` for rotation and `Eigen::Vector3d t_w_curr` for translation. Optimization is solved using Ceres.
* ROS Topics (Scan Registration --> Odometry)
  * `/laser_cloud_sharp`             feature cloud, current frame, used in current optimization
  * `/laser_cloud_less_sharp`  feature cloud, current frame, not used, saved as corner\_last
  * `/laser_cloud_flat`               feature cloud, current frame, used in current optimization
  * `/laser_cloud_less_flat`    feature cloud, current frame, not used, saved as surf\_last
  * `/velodyne_cloud_2`               full laser cloud, current frame
* ROS Topics (Odometry --> Mapping)
  * `/laser_cloud_corner_last` same as `/laser_cloud_less_sharp` in the last frame&#x20;
  * `/laser_cloud_surf_last`  same as `/laser_cloud_less_flat` in the last frame&#x20;
  * `/velodyne_cloud_3`  full laser cloud, current frame, same as `velodyne_cloud_2`
  * `/laser_odom_to_init`  latest pose estimation (odom world to robot `q/t_odom_body`)

### Transformation Updates in A-LOAM

```cpp
// laserOdometry.cpp
// two coordinates: q/t_w_curr and q/t_last_curr
// in each iteration, update global variable w_curr using local variable last_curr

// initialization
Eigen::Quaterniond q_w_curr(1, 0, 0, 0); // w, x, y, z for initialization
Eigen::Vector3d t_w_curr(0, 0, 0);
double para_q[4] = {0, 0, 0, 1}; // x, y, z, w for internal storage
double para_t[3] = {0, 0, 0};    // array format for Ceres solver
Eigen::Map<Eigen::Quaterniond> q_last_curr(para_q);
Eigen::Map<Eigen::Vector3d> t_last_curr(para_t);

void TransformToStart(pi, po) {
  // assuming no distortion and s=1 by default
  q_point_last = Eigen::Quaterniond::Identity().slerp(s, q_last_curr);
  t_point_last = s * t_last_curr;
  po = q_point_last * pi + t_point_last;
}

while (ros::ok()) {
  ros::spinOnce();
  // check message buffers and synchronize timestamps
  // run optimization twice

    // TransformToStart (cornerPointsSharp and surfPointsFlat)
    // solve for para_q and para_t using Ceres solver

  // transform update
  t_w_curr = t_w_curr + q_w_curr * t_last_curr;
  q_w_curr = q_w_curr * q_last_curr;

  // publish ros topics
}
```

```cpp
// laserMapping.cpp
// three coordinates: w_curr, wmap_wodom, wodom_curr

// initialization
double parameters[7] = {0, 0, 0, 1, 0, 0, 0};
Eigen::Map<Eigen::Quaterniond> q_w_curr(parameters);
Eigen::Map<Eigen::Vector3d> t_w_curr(parameters + 4);
// wmap_T_odom * odom_T_curr = wmap_T_curr;
// transformation between odom's world and map's world frame
Eigen::Quaterniond q_wmap_wodom(1, 0, 0, 0); // w, x, y, z for initialization
Eigen::Vector3d t_wmap_wodom(0, 0, 0);
Eigen::Quaterniond q_wodom_curr(1, 0, 0, 0);
Eigen::Vector3d t_wodom_curr(0, 0, 0);

void pointAssociateToMap(pi, po) {
  Eigen::Vector3d po = q_w_curr * pi + t_w_curr;
}

while (ros::ok()) {
  // check message buffers and synchronize timestamps
  // update q_wodom_curr and t_wodom_curr from laserOdometry ROS topic
  
  // transformAssociateToMap() as follows --> set initial guess for Ceres
  q_w_curr = q_wmap_wodom * q_wodom_curr;
  t_w_curr = q_wmap_wodom * t_wodom_curr + t_wmap_wodom;
  
  // map management: shift map boundary
  // run optimization twice

    // pointAssociateToMap (laserCloudCornerStack and laserCloudSurfStack)
    // solve for q_w_curr and t_w_curr using Ceres solver

  // transformUpdate() as follows --> save map_odom transformation
  q_wmap_wodom = q_w_curr * q_wodom_curr.inverse();
  t_wmap_wodom = t_w_curr - q_wmap_wodom * t_wodom_curr;

  // save maps using pointAssociateToMap
  // publish ros topics
}
```

## LeGO-LOAM

This project is optimized specifically for a mobile robot operating on the ground, and for multi-beam LiDAR such as Velodyne. (The original LOAM algorithm uses a 2D LiDAR operating in 3D space with a 6 DoF pose.)

### Image Projection

* Act as if we project Velodyne points back to an image. Use 2D indices and matrices to mark, label and segment Velodyne points (row, col = 16x1800), but the actual points are saved in 1D pcl::PointCloud structure in an organized way.&#x20;
* Remove outliers, find start and end orientation of current point cloud, find the row and column index and compute the depth for each point.
* Find and label ground points as follows. For points in lower scans (scan ID 0-7 in VLP-16), compute the inclination angle between current point and the point in the same column but with a level higher scan ID. If this angle is less then 10 degree, then mark both points to be ground points.
* Segment cloud as follows. Starts with label #1, for each point, compute the tangent angle between current point and the horizontal or vertical neighbor point, and assign the same label to neighbor points if this angle is less than 30 degree. In other words, this will **grow the current segment if points tend to be in the same tangent plane (wrt LiDAR center)**. Then increment the label number and proceed to the next unlabeled point.&#x20;

### Odometry and Mapping

* Most parts remain the same as in the original LOAM algorithm. One addition is that GTSAM is applied to refine the estimation output from the mapping module.&#x20;

### Loop Closure

* If there exists any keypose in the history that is closer than 7m in distance and greater than 30s in time with respect to the current keypose, we will match this keypose with the current keypose using the ICP algorithm provided in PCL. If succeeded, we add this relation/observation of two keyposes to GTSAM.&#x20;
* This is a naive method and can be improved by other algorithms such as ScanContext.

## Observations

#### Geometric features and degeneration

* Lines/edges are 1 dof (1 trans) features in 3D space; they can be used to estimate any motion that is perpendicular to the line. We have tolerance on the estimation error of this landmark along its direction.
* Planes are 3 dof (2 trans + 1 rot) features in 3D space; they can be used to estimate any motion that is (at least partially) perpendicular to the plane. We can tolerate any estimation error of this landmark inside its 3 dof plane.
* Objects are 6 dof (3 trans + 3 rot) features in 3D space; they can be used to estimate motion towards any direction. However, the tradeoff is that any estimation error of the object itself will be carried on to the estimation of the robot pose.
* As a special case of the plane, points on the ground are not useful for the translation estimation on the ground (because motion is in parallel to the plane), but they are helpful to bound any possible disturbance that happens in the z axis. This is true if the ground is rigid and flat, otherwise they are noises. (e.g., floor vs. grassland)
* Case 1: apply 0-2m filter on z axis. If no features are available from ground or ceiling, then estimation in the z axis can have large errors. The degeneration happens in the z axis since all features (lines and planes) are in parallel to the z axis.
* Case 2: infinite long corridor. Our forward motion is in parallel to all features (planes) around us. Degeneration happens along the moving direction.
* Case 3: infinite large ground surface. No landmarks around are available except for a ground plane. Any moving direction on the x-y plane is degenerative since it is in parallel to the plane.

#### Example LiDAR Odometry Module (Taken from Lego-LOAM)

* Note that in the diagram below, planar features are used to estimate tz, roll and pitch only, whereas edge features are used to estimate tx, ty, and yaw. (This odometry algorithm targets at scenarios where mobile robots operating on the ground.)

![](/files/F6sslUbcjGbwKoIHX6f3)


# Iterative Closest Point

The following shows an example implementation of ICP algorithm. It can have exactly the same performance as the ICP algorithm in PCL.

```cpp
// Implementation by Hanzhe Teng, Feb 2022
// refer to pcl/registration/impl/icp.hpp and transformation_estimation_svd.hpp
Eigen::Matrix4f ClassicICPRegistration (const PointCloudPtr& source, const PointCloudPtr& target) {
  // initialization
  int iteration = 100;
  float distance_threshold = 0.05; // icp.setMaxCorrespondenceDistance
  int cloud_size = static_cast<int> (source->size());
  Eigen::Matrix4f transformation = Eigen::Matrix4f::Identity();
  Eigen::Matrix4f final_transformation = Eigen::Matrix4f::Identity();
  pcl::PointCloud<PointT>::Ptr source_trans (new pcl::PointCloud<PointT>);

  // build K-d tree for target cloud
  pcl::search::KdTree<PointT>::Ptr kdtree (new pcl::search::KdTree<PointT>);
  kdtree->setInputCloud(target);
  std::vector<int> indices (1);    // for nearestKSearch
  std::vector<float> sq_dist (1);  // for nearestKSearch

  // repeat until convergence
  for (int t = 0; t < iteration; ++t) {
    // transform source using estimated transformation
    pcl::transformPointCloud<PointT> (*source, *source_trans, final_transformation);

    // visualize source_trans in each step if needed

    // find correspondences in target
    std::vector<std::pair<int, int>> correspondences;
    for (int i = 0; i < cloud_size; ++i) {
      kdtree->nearestKSearch(source_trans->points[i], 1, indices, sq_dist);
      if (sq_dist[0] > distance_threshold * distance_threshold) // skip if too far
        continue;
      correspondences.push_back({i, indices[0]});
    }

    // convert to Eigen format
    int idx = 0;
    Eigen::Matrix<float, 3, Eigen::Dynamic> cloud_src (3, correspondences.size());
    Eigen::Matrix<float, 3, Eigen::Dynamic> cloud_tgt (3, correspondences.size());
    for (const auto& corres : correspondences) {
      cloud_src (0, idx) = source_trans->points[corres.first].x;
      cloud_src (1, idx) = source_trans->points[corres.first].y;
      cloud_src (2, idx) = source_trans->points[corres.first].z;
      cloud_tgt (0, idx) = target->points[corres.second].x;
      cloud_tgt (1, idx) = target->points[corres.second].y;
      cloud_tgt (2, idx) = target->points[corres.second].z;
      ++idx;
    }

    // skip a few steps here for simplicity, such as
    // check convergence (if trans update < required epsilon)
    // check if cloud_src and cloud_tgt are valid (>0 or >3?)

    // solve using Umeyama's algorithm (SVD)
    transformation = Eigen::umeyama (cloud_src, cloud_tgt, false);
    final_transformation = transformation * final_transformation;
    std::cout << "it = " << t << "; cloud size = " << cloud_size << "; idx = " << idx << std::endl;
    std::cout << "current transformation estimation" << std::endl << final_transformation << std::endl;
  }

  return final_transformation;
}

```


# Generalized ICP

### The nature of the problem/algorithm

* Classic ICP uses a point-to-point distance metric, which can lead to a non-quadratic error surface, especially when the point clouds are not well aligned initially. This is because the closest point correspondence can change dramatically for different transformations, leading to a discontinuous, and hence non-quadratic, error surface.
* GICP, on the other hand, uses a plane-to-plane distance metric, which can result in a smoother and more quadratic-like error surface. This is because the plane-to-plane distance changes more smoothly with the transformation, leading to a more continuous error surface.
  * The above two statements about two methods are generated by GPT and to be verified by experiments.
* Both ICP and GICP establish error terms in the least-square forms (with or without covariance matrix as the weight), and this seems to fit GN and LM use cases. However, in practice, despite least-square error terms, the point cloud registration problem can still be highly non-linear, due to the nature of transformation matrix (in SE(3) space), and how the correspondence points are established in each iteration. Therefore, in practice, BFGS may run more robustly than GN and LM.&#x20;

### How to choose optimizer? Comparison between GN, LM, and BFGS.

* Gauss-Newton (GN): It assumes that the error surface is quadratic, which allows it to solve directly for the step that minimizes the error.
* Levenberg-Marquardt (LM): This is a modification of Gauss-Newton that introduces a damping factor, making it more robust to non-quadratic error surfaces. When the damping factor is high, LM behaves more like gradient descent, taking small, reliable steps. When the damping factor is low, LM behaves more like Gauss-Newton, taking large, confident steps. This adaptability makes LM more reliable than Gauss-Newton in practice.
  * Although both GN and LM are iterative methods for nonlinear problem, at each iteration of the state, they approximate the model using first-order Taylor expansion (i.e. linearization), then the linearized Jacobian-residual equation can be solved by a linear solver in the form of Ax=b.
  * Even in the colored point cloud registration algorithm, the color term can also be formulated as linearized, local approximation and solved by GN method.
* BFGS: This is a quasi-Newton method that approximates the Hessian matrix using gradient information instead of computing it directly. This makes it more efficient than Gauss-Newton and LM for problems *with a large number of parameters* (e.g., quadrotor motion planning). BFGS also includes a line search to find the optimal step, which can make it more robust to non-quadratic error surfaces.
  * This is a general unconstrained nonlinear optimization method that does not assume least-square error terms. It does not linearize the model (which contains first-order info only), but instead makes use of the second order information (i.e. curvature, encoded in the approximated Hessian matrix).

### Implementation Details

PCL GICP algorithm uses BFGS optimizer to perform unconstrained nonlinear optimization. It represent state in 6-by-1 vector (XYZ, RPY) as the compact form, and restores to transformation matrix when needed.

The gradient is computed separately for XYZ and RPY. For XYZ, it computes the distance residual between corresponding points, multiplied by the weights from pre-computed covariance matrices (in `df` and `fdf` functions). For RPY, it expresses the rotation matrix R as ZYX Euler angles, and then takes the derivative of the elements of the rotation matrix with respect to the Euler angles (in `computeRDerivative` function.)

Essentially, the distance metric used in plane-to-plane ICP or GICP is the Mahalanobis distance between two distributions.

### Distance Metric

The standard Mahalanobis distance is used to measure point-to-distribution distance. It is formulated as the following:

$$D(\mathbf{p}) = \sqrt{(\mathbf{p} - \boldsymbol{\mu})^\top \Sigma^{-1} (\mathbf{p} - \boldsymbol{\mu})}$$

Intuitively, Mahalanobis distance is the distance of the test point from the center of mass divided by the width of the ellipsoid in the direction of the test point.

If we extend this concept to measure the distance between two distributions (such as two planes), the formula becomes the following:

$$D(\boldsymbol{\mu}\_1, \boldsymbol{\mu}\_2) = \sqrt{(\boldsymbol{\mu}\_1 - \boldsymbol{\mu}\_2)^\top (\Sigma\_1 + \Sigma\_2)^{-1} (\boldsymbol{\mu}\_1 - \boldsymbol{\mu}\_2)}$$

This is to measure the residual between two mean points, weighted by the sum of their respective covariance matrices.

If we take into account that one of the point cloud will be transformed at each iteration, so as the covariance matrix of that point cloud, then the measure becomes:&#x20;

$$D(\boldsymbol{\mu}\_1, \boldsymbol{\mu}\_2') = \sqrt{(\boldsymbol{\mu}\_1 - \boldsymbol{\mu}\_2')^\top (\Sigma\_1 + T \Sigma\_2 T^\top)^{-1} (\boldsymbol{\mu}\_1 - \boldsymbol{\mu}\_2')}$$

This is exactly what the GICP algorithm is designed to minimize (Eq. 2 from the original paper):

$$T = \underset{T}{\arg\min} \sum\_i d\_i^{(T)^\top} \left( C\_i^B + T C\_i^A T^\top \right)^{-1} d\_i^{(T)}$$.

#### Fast-GICP Code Snippet

```cpp
double gicp_loss(const Eigen::Vector3d& mean_A, const Eigen::Matrix3d& cov_A, const Eigen::Vector3d& mean_B, const Eigen::Matrix3d& cov_B, const Eigen::Matrix3d& R, const Eigen::Vector3d& t) {
  Eigen::Vector3d d = mean_B - (R * mean_A + t);
  Eigen::Matrix3d RCR = cov_B + R * cov_A * R.transpose();
  Eigen::Matrix3d RCR_inv = RCR.inverse();
  Eigen::Vector3d RCRd = RCR_inv * d;
  double loss = d.dot(RCRd);
}
```

#### PCL Code Snippet

```cpp
// PCL registration/impl/gicp.hpp
// in function computeTransformation()
if (nn_dist_sq[0] < dist_threshold) {
  Eigen::Matrix3d& C1 = (*input_covariances_)[i];
  Eigen::Matrix3d& C2 = (*target_covariances_)[nn_indices[0]];
  Eigen::Matrix3d& M = mahalanobis_[i];
  // M = R*C1
  M = R * C1;
  // temp = M*R' + C2 = R*C1*R' + C2
  Eigen::Matrix3d temp = M * R.transpose();
  temp += C2;
  // M = temp^-1
  M = temp.inverse();
  source_indices[i] = static_cast<int>(i);
  target_indices[i] = nn_indices[0];
}
// in function df() or fdf()
for (int i = 0; i < m; ++i) {
  // The last coordinate, p_src[3] is guaranteed to be set to 1.0 in registration.hpp
  Vector4fMapConst p_src = gicp_->tmp_src_->points[(*gicp_->tmp_idx_src_)[i]].getVector4fMap();
  // The last coordinate, p_tgt[3] is guaranteed to be set to 1.0 in registration.hpp
  Vector4fMapConst p_tgt = gicp_->tmp_tgt_->points[(*gicp_->tmp_idx_tgt_)[i]].getVector4fMap();
  Eigen::Vector4f pp(transformation_matrix * p_src);
  // The last coordinate is still guaranteed to be set to 1.0
  Eigen::Vector3d res(pp[0] - p_tgt[0], pp[1] - p_tgt[1], pp[2] - p_tgt[2]);
  // temp = M*res
  Eigen::Vector3d temp(gicp_->mahalanobis((*gicp_->tmp_idx_src_)[i]) * res);
  // Increment total error
  f += double(res.transpose() * temp);
}
```

References: ChatGPT, <https://en.wikipedia.org/wiki/Mahalanobis_distance>, [Fast-GICP loss function](https://github.com/SMRT-AIST/fast_gicp/blob/87cd6288d14bd155e8b7a2144f68bb5246aecc52/include/fast_gicp/gicp/gicp_loss.hpp), [PCL GICP implementation](https://github.com/PointCloudLibrary/pcl/blob/master/registration/include/pcl/registration/impl/gicp.hpp).

### Use Point Normals to Compute Covariances

Since GICP algorithm will regularize the covariance matrices by (0.001, 1, 1) diagonal matrix, where the only information retained herein is the normal direction, it is possible to represent this covariance matrix by its point normal, and reconstruct this regularized covariance back at a later time.

To this end, we can estimate the point normals using the same method as estimating these covariance matrices, and then save this normal information in each point itself.

#### PCL Normal Estimation

In fact, the normal estimation algorithm will need to first estimate a covariance matrix (using exactly the same way as GICP), and then perform eigenvalue decomposition on this covariance matrix, to obtain its smallest eigenvalue as the normal direction. Lastly, this algorithm may flip the normal to point toward viewpoint (or sensor origin, in most cases).

#### GICP Covariance Estimation

GICP algorithm will estimate covariance matrices as usual, and then regularize the covariance matrix by (0.001, 1, 1) diagonal matrix. Two points from source and target clouds, along with their regularized covariance matrices, will be used to compute the Mahalanobis distance between each other, and contribute to the overall optimization problem as an error term.

#### Reconstruct Covariances from Normals

There are difference ways to reconstruct this. Below we compared a few methods (using Python code equivalent to original C++ implementation) and the results (in terms of Mahalanobis distance) are identical, or numerically the same.&#x20;

```python
import numpy as np

# Function to generate points on a plane
def generate_plane_points(mean, normal, n=20, noise_level=0.001):
    points = np.random.rand(n, 3)
    for i in range(n):
        points[i] = points[i] - np.dot(points[i] - mean, normal) * normal
        points[i] += np.random.normal(scale=noise_level, size=3)
    return points

# Function to calculate Mahalanobis distance
def mahalanobis_distance(mu1, mu2, sigma1, sigma2):
    cov_inv = np.linalg.inv(sigma1 + sigma2)
    delta = mu1 - mu2
    return np.sqrt(np.dot(np.dot(delta.T, cov_inv), delta))

# Function to estimate plane normal
def estimate_plane_normal(points):
    covariance_matrix = np.cov(points, rowvar=False)
    eigenvalues, eigenvectors = np.linalg.eig(covariance_matrix)
    # The normal is the eigenvector corresponding to the smallest eigenvalue
    normal = eigenvectors[:, np.argmin(eigenvalues)]
    return normal

# Regularize covariance matrix
def regularize_covariance_matrix(cov, gicp_epsilon=0.001):
    U, _, _ = np.linalg.svd(cov)
    S = np.array([1, 1, gicp_epsilon])  # Modified singular values
    cov_modified = np.dot(U, np.dot(np.diag(S), U.T))
    return cov_modified

# Reconstruct regularized covariance matrix from normal
def reconstruct_covariance_from_normal(normal, gicp_epsilon=0.001):
    # Ensure the normal is a unit vector
    normal = normal / np.linalg.norm(normal)

    # Find two vectors perpendicular to the normal
    if (normal[0] == 0) and (normal[1] == 0):
        vec1 = np.array([1, 0, 0])
    else:
        vec1 = np.array([-normal[1], normal[0], 0])
        vec1 = vec1 / np.linalg.norm(vec1)

    vec2 = np.cross(normal, vec1)
    vec2 = vec2 / np.linalg.norm(vec2)

    # Form an orthonormal basis
    U = np.column_stack([vec1, vec2, normal])

    # Apply regularization to the singular values
    S = np.array([1, 1, gicp_epsilon])  # Regularized singular values

    # Reconstruct the covariance matrix
    cov_modified = np.dot(U, np.dot(np.diag(S), U.T))
    return cov_modified

# LOCUS method to reconstruct covariance matrix from normal
def locus_two_plane_vectors_from_normal(normal):
    normal_eig = normal / np.linalg.norm(normal)
    
    if abs(normal_eig[2]) < 1e-7:
        normal_eig[2] = 1e-7
    
    vec1 = np.array([1.0, 0.0, -normal_eig[0] / normal_eig[2]])
    vec1 = vec1 / np.linalg.norm(vec1)
    
    vec2 = np.cross(normal_eig, vec1)
    vec2 = vec2 / np.linalg.norm(vec2)
    
    return 0.001 * np.outer(normal_eig, normal_eig) + np.outer(vec1, vec1) + np.outer(vec2, vec2)

# Define two planes
mean1 = np.array([0, 0, 0])
normal1 = np.array([1, 0, 0])
plane1 = generate_plane_points(mean1, normal1)

mean2 = np.array([1, 0, 0])
normal2 = np.array([-1, 0, 0])
plane2 = generate_plane_points(mean2, normal2)

# Calculate means and covariances
mu1 = np.mean(plane1, axis=0)
sigma1 = np.cov(plane1, rowvar=False)
mu2 = np.mean(plane2, axis=0)
sigma2 = np.cov(plane2, rowvar=False)

# Estimate normals using eigenvalue decomposition
normal1_estimated = estimate_plane_normal(plane1)
normal2_estimated = estimate_plane_normal(plane2)

# Calculate Mahalanobis distance before transformation
distance_before = mahalanobis_distance(mu1, mu2, sigma1, sigma2)
print("Mahalanobis distance before transformation: {:.3f}".format(distance_before))

# Regularize covariance matrices and recalculate Mahalanobis distance
sigma1_regular = regularize_covariance_matrix(sigma1)
sigma2_regular = regularize_covariance_matrix(sigma2)
distance_before_regular = mahalanobis_distance(mu1, mu2, sigma1_regular, sigma2_regular)
print("Mahalanobis distance before transformation (regularized): {:.3f}".format(distance_before_regular))

# Reconstruct covariance matrices from normals and recalculate Mahalanobis distance
sigma1_reconstructed = reconstruct_covariance_from_normal(normal1_estimated)
sigma2_reconstructed = reconstruct_covariance_from_normal(normal2_estimated)
distance_before_reconstructed = mahalanobis_distance(mu1, mu2, sigma1_reconstructed, sigma2_reconstructed)
print("Mahalanobis distance before transformation (reconstructed): {:.3f}".format(distance_before_reconstructed))

# Reconstruct covariance matrices from normals using LOCUS method and recalculate Mahalanobis distance
sigma1_reconstructed_locus = locus_two_plane_vectors_from_normal(normal1_estimated)
sigma2_reconstructed_locus = locus_two_plane_vectors_from_normal(normal2_estimated)
distance_before_reconstructed_locus = mahalanobis_distance(mu1, mu2, sigma1_reconstructed_locus, sigma2_reconstructed_locus)
print("Mahalanobis distance before transformation (reconstructed using LOCUS method): {:.3f}".format(distance_before_reconstructed_locus))
```

### Convergence Condition

The related code snippets are:

```cpp
// in gicp.hpp, function computeTransformation()
previous_transformation_ = transformation_;
try {
  rigid_transformation_estimation_(
      output, source_indices, *target_, target_indices, transformation_);
  /* compute the delta from this iteration */
  delta = 0.;
  for (int k = 0; k < 4; k++) {
    for (int l = 0; l < 4; l++) {
      double ratio = 1;
      if (k < 3 && l < 3) { // rotation part of the transform
        ratio = 1. / rotation_epsilon_;
      } else {
        ratio = 1. / transformation_epsilon_;
      }
      double c_delta = ratio *
          fabs(previous_transformation_(k, l) - transformation_(k, l));
      if (c_delta > delta) {
        delta = c_delta;
      }
    }
  }
}
if (nr_iterations_ >= max_iterations_ || delta < 1) {
  converged_ = true;
}

// default values in initialization
rotation_epsilon_(2e-3),
transformation_epsilon_ = 5e-4;
```

This is to compute the difference in the transformation matrix, represented by `delta` variable.

The algorithm is deemed converged, if no element in the rotation matrix has a difference (computed against last iteration) greater than the `rotation_epsilon_`,  and no element in the translation vector has a difference greater than the `transformation_epsilon_`.

`transformation_epsilon_` can be matched to meters directly, i.e. 5e-4 means no difference greater than 0.0005 meter.

`rotation_epsilon_` corresponds to the max difference in the elements of the 3x3 rotation matrix. For example, if a rotation matrix represents a z-axis rotation of alpha degree, then the converge condition is

rotation\_epsilon\_ = sin(alpha \* M\_PI / 180), or

alpha = arcsin(rotation\_epsilon\_) \* 180 / M\_PI.

When rotation\_epsilon\_ = 2e-3, the corresponding angle in degree is 0.11 degree. Other values can be inferred accordingly.

### Math in Optimization

* Objective function: $$T = \underset{T}{\arg\min} \sum\_i d\_i^{(T)^\top} \left( C\_i^B + T C\_i^A T^\top \right)^{-1} d\_i^{(T)}$$.
* We basically omit the derivative of the Mahalanobis matrix in the middle, and simplify it as a constant matrix denoted by M. We substitute d with its original definition into the objective function and obtain the following.
* After rearrangement:  $$f\_i(\mathbf{R}, \mathbf{t}) = (\mathbf{p}\_i - \mathbf{R} \mathbf{q}\_i - \mathbf{t})^{\top} \mathbf{M} (\mathbf{p}\_i - \mathbf{R} \mathbf{q}\_i - \mathbf{t}).$$
* Derivative of R: on: $$\frac{\mathrm{d} f\_i(\mathbf{R}, \mathbf{t})}{\mathrm{d} \mathbf{R}} = 2 (- \mathbf{q}\_i) \times \left(\mathbf{M} (\mathbf{p}\_i - \mathbf{R} \mathbf{q}\_i - \mathbf{t}) \right)^{\top}.$$
* Derivative of t: on: $$\frac{\mathrm{d} f\_i(\mathbf{R}, \mathbf{t})}{\mathrm{d} \mathbf{t}} = - 2 \mathbf{M} (\mathbf{p}\_i - \mathbf{R} \mathbf{q}\_i - \mathbf{t}).$$
* The above two derivatives are consistent with the PCL GICP implementation in C++.
* See the math notes of GICP on overleaf for more information and derivations.

### Implementations

* Original GICP authors: <https://github.com/avsegal/gicp/blob/master/bfgs_funcs.cpp>
* PCL GICP: <https://github.com/PointCloudLibrary/pcl/blob/master/registration/include/pcl/registration/impl/gicp.hpp>
* Open3D GICP: <https://github.com/isl-org/Open3D/blob/main/cpp/open3d/pipelines/registration/GeneralizedICP.cpp>
* Fast-GICP: <https://github.com/SMRT-AIST/fast_gicp/blob/master/include/fast_gicp/gicp/impl/fast_gicp_impl.hpp>

### Parameterization of Pose

* Linearization of transformation matrix
* ZYX Euler angle for rotation
* Quaternion for rotation
* Cayley–Rodrigues Parameters (in the plane adjustment paper)


# Mahalanobis Distance

In the following, we will visualize the Mahalanobis distance between two planes using a Python script.

We sample 20 points at random and organize them on two planes, centered at (0, 0.5, 0.5) and (1, 0.5, 0.5) respectively. We then rotate one of the two planes around z axis for 90 degrees, and see how the distance changes. Also, we set different noise levels on the normal direction of the planes, and see how the distance changes.

### Before Rotation, Noise Level = 0.1, M-distance = 6-9

<figure><img src="/files/WFMM5xQ3dBXKIB30zJOa" alt=""><figcaption></figcaption></figure>

### After Rotation, Noise Level = 0.1, M-distance = 2-4

<figure><img src="/files/t3ttJiWuKXIXHgE33F0l" alt=""><figcaption></figcaption></figure>

### Before Rotation, Noise Level = 0.001, M-distance = 600-900

<figure><img src="/files/LyaHbojmMcOKyLPcReMS" alt=""><figcaption></figcaption></figure>

### After Rotation, Noise Level = 0.001, M-distance = 2-4

<figure><img src="/files/x6QU0O44HyVFf1Nyy2p6" alt=""><figcaption></figcaption></figure>

Furthermore, if we set noise level to 0.01, the results are still consistent. Figures omitted.

* Before Rotation, Noise Level = 0.01, M-distance = 60-90
* After Rotation, Noise Level = 0.01, M-distance = 2-4

If we rotate the plane slightly, such as even only 10 degrees, the distance will decrease drastically.

* Before Rotation, Noise Level = 0.001, M-distance = 600-900
* After Rotation, Noise Level = 0.001, M-distance = 20-40

### Python Code

```python
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D

# Function to generate points on a plane
def generate_plane_points(mean, normal, n=20, noise_level=0.001):
    # Generate random points
    points = np.random.rand(n, 3)
    # Adjust points to lie on a given plane
    for i in range(n):
        points[i] = points[i] - np.dot(points[i] - mean, normal) * normal
        # Add some noise
        points[i] += np.random.normal(scale=noise_level, size=3)
    return points

# Function to calculate Mahalanobis distance
def mahalanobis_distance(mu1, mu2, sigma1, sigma2):
    cov_inv = np.linalg.inv(sigma1 + sigma2)
    delta = mu1 - mu2
    return np.sqrt(np.dot(np.dot(delta.T, cov_inv), delta))

# Function to plot planes
def plot_planes(plane1, plane2, ax):
    ax.scatter(plane1[:,0], plane1[:,1], plane1[:,2], color='blue', alpha=0.5)
    ax.scatter(plane2[:,0], plane2[:,1], plane2[:,2], color='red', alpha=0.5)

# Define two planes
mean1 = np.array([0, 0, 0])
normal1 = np.array([1, 0, 0])
plane1 = generate_plane_points(mean1, normal1)

mean2 = np.array([1, 0, 0])
normal2 = np.array([-1, 0, 0])
plane2 = generate_plane_points(mean2, normal2)

# Calculate means and covariances
mu1 = np.mean(plane1, axis=0)
sigma1 = np.cov(plane1, rowvar=False)

mu2 = np.mean(plane2, axis=0)
sigma2 = np.cov(plane2, rowvar=False)

# Calculate Mahalanobis distance before transformation
distance_before = mahalanobis_distance(mu1, mu2, sigma1, sigma2)
print("Mahalanobis Distance Before Transformation:", distance_before)

# Apply transformation to second plane (e.g., rotation)
theta = np.radians(90)  # Rotation angle around z axis
rotation_matrix = np.array([
    [np.cos(theta), -np.sin(theta), 0],
    [np.sin(theta), np.cos(theta), 0],
    [0, 0, 1]
])
plane2_transformed = np.dot(plane2 - mu2, rotation_matrix) + mu2

# Recalculate mean and covariance for the transformed plane
mu2_transformed = np.mean(plane2_transformed, axis=0)
sigma2_transformed = np.cov(plane2_transformed, rowvar=False)

# Calculate Mahalanobis distance after transformation
distance_after = mahalanobis_distance(mu1, mu2_transformed, sigma1, sigma2_transformed)
print("Mahalanobis Distance After Transformation:", distance_after)

# Visualization
fig = plt.figure(figsize=(10, 8))
ax = fig.add_subplot(111, projection='3d')
plot_planes(plane1, plane2_transformed, ax)
ax.set_title("Mahalanobis Distance Between Two Planes")
plt.show()
```


# Computer Science

### Algorithms

* [TheAlgorithms/C-Plus-Plus](https://github.com/TheAlgorithms/C-Plus-Plus): medium quality C++ code, but comprehensive
* [Algorithm-DataStructures](https://github.com/jakobkogler/Algorithm-DataStructures): high quality C++ code (and a few in other languages)&#x20;

#### Segment Tree

* [C++ implementation](https://github.com/jakobkogler/Algorithm-DataStructures/blob/master/RangeQuery/SegmentTree.cpp)
* [Tutorial in Chinese](https://blog.csdn.net/Yaokai_AssultMaster/article/details/79599809)
* [Tutorial on CodeForces](https://codeforces.com/blog/entry/18051)


# Computational Model

![](/files/-McaektC1vRICzPinXPk)

### Random-Access Machine (RAM)

* Unit cost for&#x20;
  * any instruction on O(log(n))-bit words
  * Read/write a single memory location from an infinite memory
* The cost measure: time complexity

variant: Parallel RAM (PRAM) model

### Fork-Join Model

* is the main model of parallel execution in the OpenMP framework
* In addition to RAM instructions, you can also use `in parallel` and `parallel for` instructions.

```cpp
mergesort(A, lo, hi):
  if lo < hi:
    mid = ⌊lo + (hi - lo) / 2⌋
    fork mergesort(A, lo, mid) // process (potentially) in parallel with main task
      mergesort(A, mid, hi)    // main task handles second recursion
    join
      merge(A, lo, mid, hi)
```

### Work-Span Model

* For all computations, draw a DAG
  * A->B means that B can be performed only when A has been finished
* **Work**: the total number of operations
* **Span** (depth): the longest length of chain

![](/files/-Mcal8btyD57on7epp8q)


# Sorting

### Selection Sort

* О(n^2) comparisons, О(n) swaps
* naive implementation for sorting

### Insertion Sort

* О(n^2) comparisons and swaps
* O(1) auxiliary space
* can run online
* stable

### Merge Sort

* O(n log(n)) by divide and conquer
* O(n) auxiliary space
* stable; deterministic

### Quick Sort

* O(n log(n)) by divide and conquer; worst case O(n) though
* O(1) auxiliary space; one of exchange sorts (cf. bubble sort)
* unstable; randomized

### Introsort

* hybrid sorting algorithm
* derived from quicksort, heapsort, and insertion sort
* unstable

### Timsort

* hybrid sorting algorithm
* derived from merge sort and insertion sort
* stable

### Distribution Sort

* O(n+m) non-comparative sorting algorithms: radix sort, bucket sort, digital sort, pigeonhole sort
* it avoids comparison by creating and distributing elements into buckets according to their radix
* the lower bound for comparative sorting algorithm is O(n log(n)); in certain cases, non-comparative sorting can do better, e.g., O(n)

### Topological Sort

* for directed acyclic graph (DAG)
* output a linear ordering of its vertices&#x20;

### Tree Sort

* builds a binary search tree from the elements to be sorted, and then traverses the tree (in-order) so that the elements come out in sorted order.
* Its typical use is sorting elements online: after each insertion, the set of elements seen so far is available in sorted order.


# Analysis

## Asymptotic Notations

* formal definition: $$f(n) = O(g(n))$$if there exists positive constants$$c$$and$$n\_0$$such that $$0 \le f(n) \le cg(n)$$for all $$n \ge n\_0$$.&#x20;
* $$\lim\_{n\to\infty} \frac{f(n)}{g(n)} \le constant$$&#x20;

|         notation        | (literal) meaning |        example 1       |         example 2        |         example 3        |
| :---------------------: | :---------------: | :--------------------: | :----------------------: | :----------------------: |
|    $$f(n) = O(g(n))$$   | $$f(n) \le g(n)$$ |     $$n = O(n^2)$$     |     $$n^2 = O(n^2)$$     |    $$n^3 \neq O(n^2)$$   |
| $$f(n) = \Omega(g(n))$$ | $$f(n) \ge g(n)$$ | $$n \neq \Omega(n^2)$$ |   $$n^2 = \Omega(n^2)$$  | $$n^3 \neq \Omega(n^2)$$ |
| $$f(n) = \Theta(g(n))$$ |  $$f(n) = g(n)$$  | $$n \neq \Theta(n^2)$$ |   $$n^2 = \Theta(n^2)$$  | $$n^3 \neq \Theta(n^2)$$ |
|    $$f(n) = o(g(n))$$   |  $$f(n) < g(n)$$  |     $$n = o(n^2)$$     |    $$n^2 \neq o(n^2)$$   |    $$n^3 \neq o(n^2)$$   |
| $$f(n) = \omega(g(n))$$ |  $$f(n) > g(n)$$  | $$n \neq \omega(n^2)$$ | $$n^2 \neq \omega(n^2)$$ |   $$n^3 = \omega(n^2)$$  |

more reference: [Orders of common functions](< https://en.wikipedia.org/wiki/Big_O_notation#Orders_of_common_functions>)

## Randomized Algorithms and Average-case Analysis

#### Examples

* the hiring problem: in average O(log(n)) instead of O(n)
* quicksort: the cost is O(n log(n)) on average
* hash table: expected O(1)
* Rabin-Karp: sub-string matching

## Amortized Analysis

Though the algorithm is deterministic, the cost of each step may be different

#### Examples

* Binary Counter and Piggy Bank
  * pay only when a bit changes from 0 to 1
* Hash Table
  * load factor better to be 1/2
  * need to resize when full
    * worst case cost: O(n)
    * amortized to O(1)
* Union-find with path-compression: O(log\* n) per union or per find
* Weight balanced tree with rebuilding
  * O(log n) cost per insertion
    * O(1) cost for rebalance per insertion
    * O(log n) cost for insertion itself

#### Three ways for amortized analysis

* Direct: count the total cost from an empty state all the way to the n-th operation O(f(n))
  * then the total cost of each operation is $$O(\frac{f(n)}{n})$$&#x20;
* Piggy bank: pay O(k) “dollars” per operation
  * show that even though some of the operations are more expensive, the total cost of all n elements are no more than O(nk)
* Potential function: design a potential function$$\Phi (s)$$for a certain state, and analyze the change of$$\Phi$$, from which we can derive the amortized cost

### &#xD;


# Complexity Classes (P, NP)

<figure><img src="/files/QGWJXY4BLuCg3AgVutme" alt="" width="563"><figcaption></figcaption></figure>

```
____________________________________________________________
| Problem Type | Verifiable in P time | Solvable in P time | Increasing Difficulty
___________________________________________________________|           |
| P            |        Yes           |        Yes         |           |
| NP           |        Yes           |     Yes or No *    |           |
| NP-Complete  |        Yes           |      Unknown       |           |
| NP-Hard      |     Yes or No **     |      Unknown ***   |           |
____________________________________________________________           V
```

Notes on `Yes` or `No` entries:

* \* An NP problem that is also P is solvable in P time.
* \*\* An NP-Hard problem that is also NP-Complete is verifiable in P time.
* \*\*\* NP-Complete problems (all of which form a subset of NP-hard) might be. The rest of NP hard is not.

### Complexity Classes for Decision Problems

* Decision problem: A problem with a yes or no answer.
  * Note that the following definitions and discussions are centered about the decision problem (or if a solution is verifiable), and do not mention if you can find the solution (or if the problem is solvable).
* P is a complexity class that represents the set of all decision problems that can be solved in polynomial time.
  * That is, given an instance of the problem, the answer yes or no can be decided in polynomial time.
* NP is a complexity class that represents the set of all decision problems for which the instances where the answer is "yes" have proofs that can be verified in polynomial time.
  * This means that if someone gives us an instance of the problem and a certificate (sometimes called a witness) to the answer being yes, we can check that it is correct in polynomial time.
  * Note that NP is for Nondeterministic Polynomial time; not non-polynomial time.
* NP-Complete: NP-Complete is a complexity class which represents the set of all problems `X` in NP for which it is possible to reduce any other NP problem `Y` to `X` in polynomial time.
  * Intuitively this means that we can solve `Y` quickly if we know how to solve `X` quickly.&#x20;
  * Precisely, `Y` is reducible to `X`, if there is a polynomial time algorithm `f` to transform instances `y` of `Y` to instances `x = f(y)` of `X` in polynomial time, with the property that the answer to `y` is yes, if and only if the answer to `f(y)` is yes.
* NP-Hard: Intuitively, these are the problems that are at least as hard as the NP-complete problems.
  * Note that NP-hard problems do not have to be in NP, and they do not have to be decision problems.
  * The precise definition here is that a problem X is NP-hard, if there is an NP-complete problem Y, such that Y is reducible to X in polynomial time.

References: <https://stackoverflow.com/questions/1857244/what-are-the-differences-between-np-np-complete-and-np-hard>


# Divide and Conquer

### Three Steps

* **Divide** the problem into multiple subproblems in smaller size.
* **Conquer** the subproblems recursively; set a straight forward way for base case.
* **Combine** the solutions to the subproblems into the solution for the original problem.

### Master Theorem

* recurrence relation: $$T(n)=a T\left({\frac{n}{b}}\right)+f(n)$$
* $$a>1$$the number of subproblems in the recursion
* $$b>1$$the factor by which the subproblem size is reduced in each recursive call
* and$$f$$is asymptotically positive (positive for sufficiently large$$n$$)
* base case: $$T(c)$$is a constant when$$c$$is a constant
* compare $$n^{\log\_{b}{a}}$$with $$f(n)$$and this will lead to three regimes (>, =, <)
  * in recursion tree, compare root with leaves and see which part dominates
* more reference: CLRS book page 94

![](/files/-MccQZWgzUmY4K5lT2aY)

### Example Code: Merge Sort

```cpp
#include <iostream>
#include <vector>

int merge(std::vector<int>& A, int start, int end, int middle, std::vector<int>& B){
  // combine 
  int i = start;
  int j = middle;
  for (int k = start; k<= end; ++k){
    if (i < middle && (j > end || A[i] <= A[j]))
      B[k] = A[i++];
    else
      B[k] = A[j++];
  }
  for (int k = start; k<= end; ++k){
    A[k] = B[k];
  }
}

int mergeSort(std::vector<int>& A, int start, int end, std::vector<int>& B){
  // divide and conquer
  if (end == start)
    return 0;
  int middle = (start + end + 1) / 2;
  mergeSort(A, start, middle - 1, B);
  mergeSort(A, middle, end, B);
  merge(A, start, end, middle, B);
}

int main(){
  int n;
  std::cin >> n; 
  std::vector<int> score(n);
  std::vector<int> temp(n);
  for (int k = 0; k < n; ++k)
    std::cin >> score[k];

  mergeSort(score, 0, n-1, temp);

  std::cout << n << std::endl;
  for (int k = 0; k < n; ++k)
    std::cout << score[k] << std::endl;

  return 0;
}
```


# Greedy Algorithm

* Greedy algorithms are applied to optimization algorithms only.
* It picks always the current best solution; no backtracking.
* Not necessarily optimal --> need to prove it.

### Optimization Problem&#x20;

* find a set of items that can&#x20;
  * meet some constraints, and
  * optimize some objective function
* examples: shortest path, minimum spanning tree
* not an optimization problem: sorting

### Prove the Optimality

To prove the optimality of a greedy strategy, we need to prove the following two parts.&#x20;

* Greedy Choice
  * find the choice that is part of SOME optimal solution
  * there may exist multiple optimal solutions; no need to prove it for ANY optimal solution
* Optimal Substructure
  * After making the first choice, the final best solution is first choice + best solution for the rest of (compatible) input
  * We can solve the same optimization problem recursively!
  * Otherwise, we find a contradiction!

### Example problems

* activity selection (pick the earliest-finish task)
* minimum spanning tree
* Huffman code
* Dijkastra's shortest path
* graph coloring
* traveling salesman

### Greedy vs. Dynamic Programming&#x20;

* Greedy: always pick the current best
* DP: pick the best based on history
* You can always use DP to solve Activity Selection problem, but it is much costly (since you have to try all other branches)

### Example Code: Easy Swaps to Sort 123

```cpp
#include <iostream>
#include <vector>
#include <algorithm>

int main(){
  int n;
  std::cin >> n; 
  std::vector<int> candies(n);
  for (int k = 0; k < n; ++k)
    std::cin >> candies[k];

  // count total number of 1s, 2s, 3s
  int sum1 = 0, sum2 = 0, sum3 = 0;
  for (int k = 0; k < n; ++k){
    if (candies[k] == 1)
      sum1++;
    else if (candies[k] == 2)
      sum2++;
    else
      sum3++;
  }

  // all 3s should be placed in the range [sum1+sum2, n-1]
  // and then leave a subproblem of 1s and 2s in the range [0, sum1+sum2-1]
  int swap = 0;
  std::vector<int> temp_queue;
  for (int k = sum1 + sum2; k < n; ++k){
    if (candies[k] != 3){
      swap++;
      temp_queue.push_back(candies[k]);
    }
  }

  // move all 1s to the front and 2s to the end
  std::sort(temp_queue.begin(), temp_queue.end());

  // in [0, sum1+sum2-1], we want to put 1s to the front and 2s to the end 
  for (int k = sum1 + sum2 - 1; k >= 0; --k){
    if (candies[k] == 3){
      candies[k] = temp_queue.back();
      temp_queue.pop_back();
    }
  }

  // finally, count how many 2s are still there in the range [0, sum1-1]
  // which is supposed to be all 1s
  for (int k = 0; k < sum1; ++k){
    if (candies[k] == 2)
      swap++;
  }

  std::cout << swap << std::endl;

  return 0;
}
```


# Dynamic Programming

* also applied to optimization problems
* make the best decision based on history
* common solution to more complicated DP problems: add more dimensions!
  * the challenging part is to find the "correct" state to solve recursively&#x20;
* common mistake: something wrong at stop condition

### Three (and a half) components

* **States**: optimal substructure
* **Decisions**: how to pick the best
* **Boundary**: stop condition&#x20;
* **Steps**: the sequence to compute (for bottom-up implementation only)

### Example Problems

* Knapsack problem and its variants
* Longest Common Subsequence (LCS)
* Edit distance
* Single-source Shortest Path (SSSP) on DAGs
* Matrix multiplication chain
* DP on trees (no-boss party)
* DP for games (NIM, Tic-Tac-Toe, Go)
* Longest Increasing Subsequence (LIC)
* Variant of Activity Selection (longest total length, largest total value, limited size)
* Line-breaking problem in Latex

#### Knapsack Problems

* unbounded knapsack
* 0/1 knapsack
* Multiple knapsack problem: item$$j$$can be used$$x\_j$$times
  * make $$x\_j$$copies of item$$j$$&#x20;
  * add one more dimension$$k$$with constraint$$k \le x\_j$$(the state will be$$S\_{ijk}$$)
* Some of them cannot be added together
  * for arbitrary constraint, it can be NP-hard
  * for 2-item case, if we can pick only one of (a1, a2), we can consider the pair as j
    * manually write down three possible states \[a1, a2, none] in the recurrence
* Each item has more than one dimension (e.g., both weight and volume)
  * just add one more dimension for volume
* Dependencies between items (HW Problem C?)
  * depends on the structure (tree, DAG, etc.), we can still run DP on it
  * probably need to use memorization as well

### Implementation

* Top-down method: solve recursively
  * easy to implement
    * straightforward when the recurrence relation has been established
  * compute upon needed
  * fast if don't go too deep in the recursion tree
    * will consume too much memory on stack and become slow
* Bottom-up method: non-recursive
  * hard to implement (need to be extremely careful about prerequisites)
    * data must be ready before get used; can easily make mistakes here
    * will add the fourth component to DP problem: **steps** (the sequence to compute)
  * run on each level sequentially
  * fast if all computed states are useful/needed
    * in some cases, computed states are not used later on, which is a waste of time

### Design a DP algorithm

1. find optimal substructure and represent the subproblems as states
   * find the critical properties related to subproblems
   * if it doesn't work, consider add one more dimension to the state
2. write down the recurrence between the states, and carefully check the boundary cases
3. write down the DP algorithm according to the recurrence
   * recursive or non-recursive implementation
   * memorization

### Summary&#x20;

#### State: what is the index of your DP table?

* It can describe the current stage
  * Knapsack: f\[i, j]: first i items with weight j
  * LCS/Edit distance: f\[i, j]: first i characters in X and first j characters in Y
  * LIS: f\[i]: LIS of the first i elements
  * Activity selection: f\[i]: first i activities
* It can be the amount of “resource” used
  * Knapsack: f\[i, j]: first i items with weight j
  * Activity selection: f\[i, j]: first i activities up to time j
  * Tree DP: f\[i, j]: i’th subtree with j elements selected
* It can be the current “situation”
  * DP for games: f\[i]: i describes the current “chessboard”
* It can be an interval
  * Matrix multiplication chain: f\[i, j]: best result of processing the i-th element to the j-th element

#### Boundary: stop condition; no dependence on any other states

* f\[0], f\[0, 0]
* f\[0, j], f\[i, 0]
* f\[i, i]
* f\[s] for a certain s (starting point)
* f\[i] = xxx if i = yyy

#### Decision: compute the current state based on previous states

* Consider all possibilities: list all of them, use optimal substructure
* Take min/max of them
* Save result in DP table

#### Tricks

* When you have more constrains, add a dimension
  * Knapsack f\[i, j, k, l, ..]: first i items with weight j, volume k, price l …
* By using different variables in the state, we can get different algorithms
  * Activity selection: using f\[i] as “best result using first i activities with i selected”. O(n^2) time
  * Activity selection: using f\[i, j] as “best result using first i activities up to time j”. O(nt) time
* When the ordering is hard to decide, consider using memorization

## Example Code: Ski - 2D LIS

```cpp
#include <iostream>
#include <vector>
#include <array>
#include <cmath>
#include <functional>


int main(){
  int row, col;
  std::cin >> row >> col;
  std::vector<std::vector<int>> matrix(row+2);
  for (int i = 0; i <= row+1; ++i){
    std::vector<int> column(col+2);
    matrix[i] = column;
  }
  std::vector<std::vector<int>> ans = matrix;

  for (int i = 1; i <= row; ++i)
    for (int j = 1; j <= col; ++j)
      std::cin >> matrix[i][j];

  // first find all local maximum (greater than four neighbors)
  std::vector<std::array<int, 2>> peak;
  for (int i = 1; i <= row; ++i)
    for (int j = 1; j <= col; ++j)
      if (matrix[i][j] >= matrix[i-1][j] && matrix[i][j] >= matrix[i+1][j] && 
          matrix[i][j] >= matrix[i][j-1] && matrix[i][j] >= matrix[i][j+1])
        peak.push_back({i, j});

  // define recursive lambda function
  const std::vector<std::vector<int>> mat = matrix;
  std::function<int(int, int)> ski = [&mat, &ans, &ski, row, col](int i, int j)->int{
    if (i == 0 || j == 0 || i > row || j > col) return 0;
    if (ans[i][j] != 0) return ans[i][j];
    int best = 1;
    if (mat[i][j] > mat[i-1][j]) best = std::max(best, ski(i-1, j) + 1);
    if (mat[i][j] > mat[i+1][j]) best = std::max(best, ski(i+1, j) + 1);
    if (mat[i][j] > mat[i][j-1]) best = std::max(best, ski(i, j-1) + 1);
    if (mat[i][j] > mat[i][j+1]) best = std::max(best, ski(i, j+1) + 1);
    return ans[i][j] = best;
  };

  // then run DP from each local maximum
  int max_length = 0;
  for (auto p : peak){
    int length = ski(p[0], p[1]);
    if (length > max_length)
      max_length = length;
  }

  std::cout << max_length << std::endl;

  return 0;
}
```


# Tree

## Abstract Data Type (ADT)

A mathematical model for data types:

* ~~How to store data~~
* What query to support? (lookup, findMin, findSum, …)
* What update to support? (insertion, deletion, filter, multi\_insert, delete\_min, …)
* ~~Algorithms for the operations~~

### Examples of ADT

* FIFO Queue
* Deque (double-ended queue)
* Stack
* Priority queue
* Ordered set/map
* Unordered set/map

## Winning Tree

* An implementation of priority queue
* Easy to implement, same bound as binary heap
* Can also be used as a **static** version of search tree

#### Similar concepts

* Also Winner Tree, Tournament Tree
* Segment Tree (w/o insert, delete operations).&#x20;
* Fenwick tree or binary indexed tree (only used to efficiently calculate prefix sums)

#### Use case (as priority queue)

* Huffman coding: need `extract_min` and `insertion`
* Dijkstra's algorithm and Prim's algorithm: `extract_min` and `add/update` &#x20;

### Implementation

* store all elements at the leaves (N nodes)
* each internal node is a competition (N-1 nodes)
* insertion, deletion, update operations at O(log n) time; the bound is tight
* can also use an array to store it
  * two children of A\[i]: A\[2i] and A\[2i+1]  (start from 1 can make it easier, but can also start from 0)
  * a complete binary tree
* a good alternative to binary heap
  * same asymptotical bound for insertion/deletion/update/extract\_min
  * easy to implement
  * takes more space, and is slightly slower in practice
  * when combining with augmentation, can be used to implement almost all data structures

#### References

* [TheAlgorithms/C-Plus-Plus](https://github.com/TheAlgorithms/C-Plus-Plus): medium quality C++ code, but comprehensive
* [Algorithm-DataStructures](https://github.com/jakobkogler/Algorithm-DataStructures): high quality C++ code (and a few in other languages)&#x20;
* Segment Tree
  * [C++ implementation](https://github.com/jakobkogler/Algorithm-DataStructures/blob/master/RangeQuery/SegmentTree.cpp)
  * [Tutorial in Chinese](https://blog.csdn.net/Yaokai_AssultMaster/article/details/79599809)
  * [Tutorial on CodeForces](https://codeforces.com/blog/entry/18051)

### Augmented trees

* Range-related queries: 1D range max/min/sum
  * Extract-Min is a special range query (on the entire range)
* We can augment the winning tree according to designated tasks
  * store different field for different query
  * often augment multiple fields at the same time
* If we pre-sort all elements or the key range is fixed (e.g., entire range)
  * winning tree can be used as a static search tree
* For floating-point range query, we can discretize it and map it to integers

![](/files/-Mcc9pfJB-MgMFNXeAiM)

![](/files/-Mcc9y22UFhLdQvM4Tcm)

## Example Code: Greedy & Winner Tree (code to be improved)

```cpp
#include <iostream>
#include <vector>
#include <array>
#include <list>
#include <set>
#include <map>
#include <unordered_set>
#include <unordered_map>
#include <algorithm>
#include <cmath>
#include <limits>
#include <functional>

struct node {
  int key_min; 
  int key_max;
  int value_min;
  int value_max;
  int idx;
};

// winner tree implementation
// faster, passed 8/10 cases
// the way to remove/update "used" nodes can be faster?
int main(){
  int n, m;
  std::cin >> n >> m; 
  std::vector<int> instructor(n);
  for (int i = 0; i < n; ++i)
    scanf("%d", &instructor[i]);
  std::sort(instructor.begin(), instructor.end());

  // construct (static) winner tree
  std::vector<node> tree(2*m);
  for (int i = m; i < 2*m; ++i) {
    scanf("%d %d", &tree[i].key_max, &tree[i].value_min); // key, value --> student start day, end day
    tree[i].idx = i;
    tree[i].key_min = tree[i].key_max;
    tree[i].value_max = tree[i].value_min;
  }
  for (int i = m-1; i > 0; --i) {  // leave idx 0 unused
    tree[i].key_min = std::min(tree[2*i].key_min, tree[2*i+1].key_min);       // save min key
    tree[i].key_max = std::max(tree[2*i].key_max, tree[2*i+1].key_max);       // save max key
    tree[i].value_min = std::min(tree[2*i].value_min, tree[2*i+1].value_min); // save min value
    tree[i].value_max = std::max(tree[2*i].value_max, tree[2*i+1].value_max); // save max value
    tree[i].idx = tree[2*i].value_min < tree[2*i+1].value_min ? tree[2*i].idx : tree[2*i+1].idx; // save min value idx
  }

  // this recursive function is the critical part
  std::function<void(int,int,int,int&,int&)> range_min = [&tree, &range_min, &m](int i, int key_right, int value_left, int& min_value, int& idx){
    if (i >= 2*m) return;
    if (tree[i].key_min > key_right || tree[i].value_max < value_left) return;
    if (tree[i].key_max <= key_right && value_left <= tree[i].value_min){
      if(min_value > tree[i].value_min){
        min_value = tree[i].value_min;
        idx = tree[i].idx;
      }
      return;
    }
    range_min(i*2, key_right, value_left, min_value, idx);
    range_min(i*2+1, key_right, value_left, min_value, idx);
  };

  auto tree_update = [&tree](int i, int new_value){
    tree[i].value_min = new_value;
    tree[i].value_max = new_value;
    while (i > 1){
      i = i / 2;
      tree[i].value_min = std::min(tree[2*i].value_min, tree[2*i+1].value_min); // save min value
      tree[i].value_max = std::max(tree[2*i].value_max, tree[2*i+1].value_max); // save max value
      tree[i].idx = tree[2*i].value_min < tree[2*i+1].value_min ? tree[2*i].idx : tree[2*i+1].idx; // save min value idx
    }
  };
  
  int max_num_students = 0;
  for (auto& ins_day : instructor) {    // O(n)
    int earliest_stu_end_day = std::numeric_limits<int>::max();
    int earliest_idx = -1;
    range_min(1, ins_day, ins_day, earliest_stu_end_day, earliest_idx); // O(log(m))
    if (earliest_idx == -1) continue;
    tree_update(earliest_idx, std::numeric_limits<int>::max());  // O(log(m))
    max_num_students++;
  }
  
  std::cout << max_num_students << std::endl;

  return 0;
}
```


# Graph

## Graph Representation

![](/files/-Mcf6Ymr02o1OaYHVQBb)

## Graph Problems and Algorithms

![](/files/-Mcf9Fdocg84G5Q_Gg4n)

In addition, we also have planar graphs, dynamic graphs, etc.

### Traversal-based Algorithms Review

* Breadth-first Search (BFS)
* Depth-first Search (DFS)
* Topological Sort
* Graph Connectivity
  * for each vertex, if not visited, run BFS/DFS on it
* Algorithms based on DFS
  * Bi-connectivity, articulation points, bridges (CLRS pp. 621-622)
  * Topological sort (CLRS pp. 612-614)
  * Strongly connected components (CLRS pp. 615-618)

## Minimum Spanning Tree (MST)

* undirected weighted graph
* find n-1 edges that connected all vertices with minimum total weights

#### Light-edge property

* The lightest edge in a cut must be in the MST

#### Prim's Algorithm

* start from an arbitrary vertex, add the lightest connected edge to the tree, and repeat
* this is a greedy algorithm
* implementation can be array-based, binary heap, or Fibonacci heap (CLRS Section 19)

![](/files/-McfPe4u6vzUY6nM6hZe)

#### Kruskal’s Algorithm

* scan edges by weight from lowest to highest, add the edge that doesn't introduce cycle, and repeat
* this is a greedy algorithm
* data structure: weighted edge list O(m log m) = O(m log n)

![](/files/-McfQJy-kazTgwZy5ljc)

```c
// linked forest + path compression 
find(x)     { return label[x] = x==label[x] ? x : find(label[x]); }
union(x, y) { if (find(x) != find(y)) label[label[y]] = label[x]; }
test(x, y)  { return (find(x) == find(y)); }
```

## Single-Source Shortest Paths (SSSP)

* undirected graph, or directed acyclic graph (DAG)
* find a shortest path from a given source vertex u to each vertex v

#### Bellman-Ford Algorithm

* dynamic programming algorithm O(nm)
* can work for negative weight
* $$D\_{i,k} = \min \begin{cases} D\_{i, k-1} \ \min\_{(j, i)\in E} {D\_{j, k-1} + w(j, i)}\end{cases}$$&#x20;

```cpp
// Bellman-Ford Algorithm
for i=1 to n do
  D[i]=MAX
D[s]=0
for k=1 to n-1 do
  for each (i,j) in E do
    if (D[i]+w(i,j)<D[j])
      D[j]=D[i]+w(i,j), from[j]=i
// optional optimization: if no distance is updated, break
```

#### Dijkstra's Algorithm

* start from an arbitrary node, expand the neighbor with the lowest cost, and repeat
* greedy algorithm (similar to Prim's)
* for positive weight only
* implementation: array-based O(n^2), binary heap O((m+n) log n), or Fibonacci heap (m + n log n)

#### All-Pairs Shortest Paths (APSP)

* find a shortest path from u to v for every pair of vertices u and v
* although we can solve this by running a single-source algorithm once from each vertex, we usually can solve it faster (e.g., by Floyd-Warshall algorithm)

![Relax operation: update (u, v) path by weight w if better (CLRS pp. 648)](/files/-Mck-umC7n33WCPmo4AM)

## Strongly Connected Components (SCC)

* directed graph
* strongly connected: if every vertex is reachable from every other vertex
* find a partition such that each subgraph is strongly connected

#### Tarjan’s algorithm

* is a DFS-based linear-time algorithm O(n + m)
* kind of simple to implement, but counter-intuitive
* hard to parallel
  * DFS is P-Complete (DFS is inherently sequential \[Reif 1985])

#### Reachability-based algorithm

* find SCC by reachability queries O(m log n)
  * what is the set of vertices that can be reached by a given vertex v
  * which vertices can reach v
  * the intersection of two sets form an SCC
* can be parallelized (BGSS 2016, Yan Gu, [code](https://github.com/ParAlg/gbbs/tree/master/benchmarks/StronglyConnectedComponents/BGSS16-Filtering))
* [benchmark suite for graph algorithms](https://github.com/ParAlg/gbbs)

![](/files/-McmcyejMVYpEGfocY4p)

## Bipartite Graph Matching

* **bipartite graph**: vertex set V can be partitioned into two subsets V1 and V2 such that each edge has one endpoint in V1 and the other endpoint in V2
* **matching**: a subset of the edges for which every vertex belongs to at most one of the edges
* **maximum matching**: a matching with the maximum number of edges
* **perfect matching**: a matching in which every vertex is matched

#### augmenting path

* a structure such that reversing path direction can get one more matched edge
  * unmatched vertex --- matched vertex --- ... --- matched vertex --- unmatched vertex&#x20;
* how to find:
  * reverse the matched edges
  * start from an unmatched vertex, and try to find a path to another unmatched vertex

#### Hungarian algorithm

* find the maximum matching for bipartite graphs in O(nm) time
* find an augmenting path, reverse direction, get one more matched edge, and repeat

```cpp
// a DFS-based implementation
// g[i][j] is the graph, p[j]=i stores the matched edge, 
// and b[j] is a temp boolean array
bool find(i)
  for j=1 to m do
    if g[i][j] == 1 and b[j] == false
      b[j] = true
      if p[j] == -1 or find(p[j])
        p[j]=i, return true
return false
Hungarian()
  p[1..n] = -1
  for i=1 to n do
    b[1..n] = false, find(i)
```

![](/files/-Mcmpb2Mav3E0wFx-4PH)

## Maximum Flow (Flow Network)

* directed graph, with non-negative weight/capacity
* find the max-flow from the source s to the sink t

#### Ford-Fulkerson method

* greedy algorithm
* the notion of augmenting path
  * here we overload the notion of augmenting path on a general directed graph (instead of bipartite)
  * it is called augmenting path because once found, we can get additional flow from source to sink
* method: keep finding augmenting paths in the residual graph
  * start with finding an arbitrary path from s to t&#x20;
  * reverse it, resulting in a residual graph
  * find a new path from s to t on the residual graph, and reverse it to generate a new residual graph
  * repeat until no paths can be found, then the sum of capacities of all found paths is the max-flow
* It is called "method" but not "algorithm" because how to find augmenting path is missing
  * depending on the actual method to find augmenting path, the algorithm efficiency can differ

![](/files/-Mcmrq1nJ_3D0o2WSNR4)

![](/files/-McmtQwq4s9oHgu4G1KI)

#### Ways to find augmenting paths in a general directed graph

![](/files/-McmpCkINEd2RC7Uavsq)

#### Dinic's Algorithm

* new concepts: level graph
  * construct level graph: BFS from source to every node and compute hops
  * then find augmenting paths on level graphs
* proposed by Dinic \[1970]; fastest in practice O(n^2 m)
* on bipartite graph, this algorithm is referred to as Hopcroft-Karp algorithm \[1973]

### Related Discussions on Flow Network

#### Min-cost Flow Problem

* in a flow network, each edge also has a cost
* the goal is to maximizing the flow while minimizing the cost

#### Relationship to Linear Programming

* most combinatorial algorithms can be represented in a LP form (and solved by LP solver efficiently)
* we can convert a network flow problem into a LP problem
  * by formulating edges into linear constraints
* we can also convert a LP problem into a network flow problem
  * by adding a virtual node, for example
  * the flow property can promise the equality constraints

#### The dual of max matching: min vertex cover

* vertex cover: a set of vertices that includes at least one endpoint of every edge
* in any bipartite graph, maximum matching = minimum vertex cover  \[Konig and Egercary 1931]

#### The dual of max flow: min cut

* cut: a subset of edges that disconnects the source and the sink
* minimum cut: the cut with minimum sum of edge capacities
* in a flow network, maximum flow = minimum cut  \[Menger 1927]

## Summary

* MST algorithms: Prim’s, Kruskal’s
* SSSP algorithms: Dijkstra’s, Bellman-Ford
* SCC algorithms: Tarjan’s algorithm, BGSS algorithm
* Bipartite graph matching: Hungarian algorithm
* Max flow: Ford-Fulkerson method

## Example Code: Graph and BFS

```cpp
#include <iostream>
#include <vector>
#include <array>
#include <list>
#include <queue>
#include <stack>
#include <set>
#include <map>
#include <unordered_set>
#include <unordered_map>
#include <cmath>
#include <functional>


class MyGraph { 
  private:
    // indices of vertices may not be consecutive
    std::set<int> vertices;                  // std::set, std::map -> O(log(n)) using red-black tree
    std::map<int, std::list<int>> edgeLists; // std::unordered_set, unordered_map -> O(1) using hashing
                                             // switching to unordered_set/map can speed up 600ms --> 400ms
  public:
    MyGraph(){};
    ~MyGraph(){};
    void addEdge(int src, int dest);
    void printGraph();
    void BFS(int startVertex, int& endVertex, int& depth);
};

void MyGraph::addEdge(int src, int dest) {
  vertices.insert(src);
  vertices.insert(dest);
  edgeLists[src].push_back(dest);
  edgeLists[dest].push_back(src);  // undirected graph 
}

void MyGraph::printGraph(){
  std::cout << "vertices:" << std::endl;
  for (const auto& v : vertices)
    std::cout << v << " ";
  std::cout << std::endl;
  std::cout << "edges in adjacency lists:" << std::endl;
  for (const auto& v : vertices){
    std::cout << v << ": ";
    for (const auto& e : edgeLists[v])
      std::cout << e << " "; 
    std::cout << std::endl;
  }
  std::cout << std::endl;
}

void MyGraph::BFS(int startVertex, int& endVertex, int& depth) {
  std::unordered_map<int, bool> visited (vertices.size());  // default to false
  std::unordered_map<int, int> distance (vertices.size());
  std::vector<int> visited_list;
  std::queue<int> queue;
  visited[startVertex] = true;
  distance[startVertex] = 0;
  queue.push(startVertex);
  while (!queue.empty()) {
    int currVertex = queue.front();
    queue.pop();
    visited_list.push_back(currVertex);
    for (const auto& adjVertex : edgeLists[currVertex])
      if (!visited[adjVertex]) {
        visited[adjVertex] = true;
        distance[adjVertex] = distance[currVertex] + 1;
        queue.push(adjVertex);
      }
  }
  endVertex = visited_list.back();
  depth = distance[endVertex];
}

int main(){
  int n;
  std::cin >> n; 
  MyGraph graph;
  int i, j;
  for (int k = 0; k < n-1; ++k){
    std::cin >> i >> j;
    graph.addEdge(i, j);
  }

  // run BFS twice to find the diameter of the graph
  // can be proved by triangle inequality
  int startVertex, endVertex, depth;
  startVertex = i;
  graph.BFS(startVertex, endVertex, depth);

  startVertex = endVertex;
  graph.BFS(startVertex, endVertex, depth);

  std::cout << (depth+1)/2 << std::endl;  // ceiling

  return 0;
}

```


# Computer Vision


# Camera Models

## Pinhole Model

![](/files/cim7jmxwxFccZOE9tiBW)

Let point $$P = \[X, Y, Z]$$ located at the object in 3D space, point $$P' = \[X', Y', Z']$$ located on the imaging plane in 3D space, point $$P'' = \[u, v]$$ located on the image in pixel coordinates.&#x20;

Since most modern digital cameras will automatically flip the image to its upright position for us, we can move the imaging plane to the front side as well (same side as the object). Then we can make math easier (replace the -1 with 1) and have $$X' = f \frac{X}{Z}$$and $$Y' = f \frac{Y}{Z}$$ according to the similar triangle.&#x20;

The relation between three reference frames (O, O', and O'') is the following.&#x20;

$$
\begin{align\*} u &= \alpha X' + c\_x = f\_x \frac{X}{Z} + c\_x \ v &= \beta Y' + c\_y = f\_y \frac{Y}{Z} + c\_y \end{align\*}
$$

In matrix form, we can obtain the camera intrinsic matrix $$K$$.&#x20;

$$
\begin{equation\*}
Z\left(\begin{array}{l}
u \\
v \\
1
\end{array}\right)=\left(\begin{array}{ccc}
f\_{x} & 0 & c\_{x} \\
0 & f\_{y} & c\_{y} \\
0 & 0 & 1
\end{array}\right)\left(\begin{array}{l}
X \\
Y \\
Z
\end{array}\right) \stackrel{\text { def }}{=} \boldsymbol{K} \boldsymbol{P}
\end{equation\*}
$$

## MEI Model or UCM Model

* Paper: [Single View Point Omnidirectional Camera Calibration from Planar Grids](http://www.robots.ox.ac.uk/~cmei/articles/single_viewpoint_calib_mei_07.pdf)

## References

* [The Double Sphere Camera Model](https://ieeexplore.ieee.org/abstract/document/8491007), Daniel Cremers group, 3DV 2018
  * Very good discussions and summary of camera models; read this paper first!
* [OpenCV Documentation: Camera Calibration and 3D Reconstruction](https://docs.opencv.org/2.4.8/modules/calib3d/doc/camera_calibration_and_3d_reconstruction.html)
* <https://en.wikipedia.org/wiki/Pinhole_camera_model>
* 14 Lectures in Visual SLAM


# Distortion

Assuming the camera model is pinhole, according to Brown's distortion model, there are two types of common distortions: **radial** distortion and **tangential** distortion.

### Radial distortion

* The most commonly encountered distortions are radially symmetric, due to the symmetry of a photographic lens. These radial distortions can be classified as either barrel distortions or pincushion distortions.&#x20;
  * Barrel distortion: image magnification decreases with distance from the optical axis.
  * Pincushion distortion: image magnification increases with distance from the optical axis.

![](/files/UbGzu757iJpXlZiNyxq1)

* Mathematically, barrel and pincushion distortion are **quadratic**, meaning they increase as the square of distance from the center. We can model it with three parameters: $$k\_1, k\_2, k\_3$$.

$$
x\_{\text {distorted }}=x\left(1+k\_{1} r^{2}+k\_{2} r^{4}+k\_{3} r^{6}\right)
\ y\_{\text {distorted }}=y\left(1+k\_{1} r^{2}+k\_{2} r^{4}+k\_{3} r^{6}\right)
$$

### Tangential distortion

* Tangential distortion can happen when the lens is not fully parallel to the image plane.

![](/files/QiHAmP3RJ9K0UpWqL079)

* Mathematically, we can model this distortion with two parameters: $$p\_1, p\_2$$.

$$
x\_{\text {distorted }}=x+2 p\_{1} x y+p\_{2}\left(r^{2}+2 x^{2}\right) \ y\_{\text {distorted }}=y+p\_{1}\left(r^{2}+2 y^{2}\right)+2 p\_{2} x y
$$

#### References

* <https://en.wikipedia.org/wiki/Distortion_(optics)>
* 14 Lectures in Visual SLAM


# Motion Models

Motion Models

* translation
* rotation
* scale/similarity
* shear
* affine&#x20;
* perspective

Homogeneous Transformation

Perspective Transformation


# Shutter

### Global Shutter&#x20;

* all pixels in a frame are exposed to light at the same time
* mostly supported by CCD image sensor (one read per frame)

### Rolling Shutter

* pixels are read and stored line-by-line in a frame
* mostly happens as the side effect of CMOS sensors

#### Example image in the presence of rolling shutter effect.

<div align="left"><img src="/files/-MGAhqmg_yewn4rOZBRB" alt=""></div>

#### Example video in the presence of rolling shutter effect (also called "jelly" effect).

![](/files/-LwZTo9ow6VwVKtKhkM8)

Three factors contributing to this jelly effect:

* The camera is a rolling shutter camera
* The camera is rigidly attached to a quadrotor (without gimbal)
* The quadrotor frame by itself has high frequency vibration (without damping design)


# Image Sensors

CCD and CMOS are two types of image sensors commonly used in modern digital cameras. They both use arrays of silicon pixels to detect light.

### Charge-coupled device (CCD)

* commonly seen in professional photography and telescopes
* expensive
* mostly global shutter (all pixels exposed to light at the same time)
* less visual noise and distortion

### Complementary metal–oxide–semiconductor (CMOS)

* commonly seen in small-scale devices like mobile phone
* inexpensive
* mostly rolling shutter (pixels exposed to light line-by-line)
* noisy


# Epipolar Geometry

Assuming that cameras satisfy pinhole model, we have the following geometry constraints from two perspective views.&#x20;

![](/files/5gxHsglxSLySguQal1Ue)

* Epipoles or epipolar points: $$e\_1, e\_2$$
* Epipolar lines: $$l\_1, l\_2$$
* Epipolar plane: the plane formed by optical centers $$O\_1, O\_2$$ and point $$P$$

### Fundamental Matrix

Suppose that the transformation from frame $$O\_1$$ to frame $$O\_2$$ can be described by a rotation matrix$$\boldsymbol{R}$$ and a 3D translation vector $$\boldsymbol{t}$$. Let $$\boldsymbol{P\_1}$$ denote the position of point $$P$$ in frame $$O\_1$$, and $$\boldsymbol{P\_2}$$ the position of point $$P$$ in frame $$O\_2$$. We then have the relation

$$
\begin{equation} \boldsymbol{P\_2} = \boldsymbol{R} \boldsymbol{P\_1} + \boldsymbol{t}  \end{equation}
$$

To derive the beautiful and concise epipolar constraint, which reflects the fact that $$O\_1, O\_2$$ and $$P$$ are co-planar, we first left-multiply both sides of (1) by$$\[\boldsymbol{t}]$$ to obtain

$$
\begin{equation} \[\boldsymbol{t}] \boldsymbol{P\_2} = \[\boldsymbol{t}] \boldsymbol{R} \boldsymbol{P\_1} + \[\boldsymbol{t}] \boldsymbol{t}  \end{equation}
$$

where$$\[\boldsymbol{t}]$$ is the [bracket notation](https://en.wikipedia.org/wiki/Cross_product#Conversion_to_matrix_multiplication) (i.e. express cross product as a skew-symmetric matrix) of vector $$\boldsymbol{t}$$. The term $$\[\boldsymbol{t}] \boldsymbol{t}$$ is always zero and can be crossed out. We then left-multiply both sides of (2) by $$\boldsymbol{P\_2}^{\mathrm{T}}$$ to obtain

$$
\begin{equation} \boldsymbol{P\_2}^{\mathrm{T}} \[\boldsymbol{t}] \boldsymbol{P\_2} = \boldsymbol{P\_2}^{\mathrm{T}} \[\boldsymbol{t}] \boldsymbol{R} \boldsymbol{P\_1}   \end{equation}
$$

The term$$\boldsymbol{P\_2}^{\mathrm{T}} \[\boldsymbol{t}] \boldsymbol{P\_2}$$ is always zero, because $$\boldsymbol{P\_2}$$ is perpendicular to$$\[\boldsymbol{t}] \boldsymbol{P\_2}$$ and the dot product of the two is always zero. Finally, we obtain the epipolar constraint

$$
\begin{equation}
\boldsymbol{P\_2}^{\mathrm{T}} \[\boldsymbol{t}] \boldsymbol{R} \boldsymbol{P\_1}   = 0 \end{equation}
$$

Let $$\boldsymbol{F} = \[\boldsymbol{t}] \boldsymbol{R}$$ be the Fundamental Matrix, we can rewrite the epipolar constraint in a concise form

$$
\begin{equation}
\boldsymbol{P\_2}^{\mathrm{T}} \boldsymbol{F} \boldsymbol{P\_1}   = 0 \end{equation}
$$

The epipolar constraint in this form relates the coordinates of the same point in two references frames.

### Essential Matrix

Recall the pinhole camera model, we can project point $$P$$ onto the image planes in two views by&#x20;

Taking the vector product with $$t$$, followed by the dot product with $$p\_2$$ we obtain $$p\_2^{T} \[t]\_{\times}Rp\_1 = 0$$.

$$
\begin{equation\*}
\boldsymbol{E}=\boldsymbol{t}^{\wedge} \boldsymbol{R}, \quad \boldsymbol{F}=\boldsymbol{K}^{-\mathrm{T}} \boldsymbol{E} \boldsymbol{K}^{-1}, \quad \boldsymbol{x}*{2}^{\mathrm{T}} \boldsymbol{E} \boldsymbol{x}*{1}=\boldsymbol{p}*{2}^{\mathrm{T}} \boldsymbol{F} \boldsymbol{p}*{1}=0
\end{equation\*}
$$

&#x20;its normalized image coordinates (i.e. $$x\_1 = K^{-1} p\_1$$), and the


# Multiple-View Geometry

### Chapter 1: Introduction

#### Projective Geometry

* preserve straightness
* Projective in homogeneous coordinates: both$$(x, y, 1)$$and$$(x, y, 0)$$are defined

#### Affine Geometry

* In Projective Space, we single out a particular line and call it the line at infinity.
* With this line at infinity, we are able to define parallelism, and equal length of two intervals.

#### Euclidean Geometry

* In Affine Geometry, we further single out two circular points in the line at infinity.&#x20;
* Two circular points: $$(1, ±i, 0)^\mathrm{T}$$which satisfy a pair of real equations: $$x^2 + y^2 = 0; w = 0$$.&#x20;
* With two circular points, we are able to define angle and length ratios.
* Representation: $$(x, y)$$


# Datasets


# RGB-D Datasets

## RGB-D Datasets

|   Name  | Depth Image | Color Image | Ground Truth |
| :-----: | :---------: | :---------: | :----------: |
| 3DMatch |      X      |      X      |  camera pose |
|   TUM   |             |             |              |

### 3DMatch

* paper: 3DMatch: Learning Local Geometric Descriptors from RGB-D Reconstructions, CVPR 2017
* website: [https://3dmatch.cs.princeton.edu/](https://3dmatch.cs.princeton.edu/#rgbd-reconstruction-datasets)
* notes: deep learning

### UW

* paper: A Large-Scale Hierarchical Multi-View RGB-D Object Dataset, ICRA 2011
* website: <http://rgbd-dataset.cs.washington.edu/dataset.html>

### TUM

* website: <https://vision.in.tum.de/data/datasets/rgbd-dataset/download>

### OpenLORIS

* weblink: <https://shimo.im/docs/HhJj6XHYhdRQ6jjk/read>

### Stanford Bunny

* website: <http://graphics.stanford.edu/data/3Dscanrep/>

### Berkley Angle

* website: <http://graphics.berkeley.edu/resources/ScannedModels/index.html>

## RGB-D Reconstruction Datasets organized by [3DMatch](http://3dmatch.cs.princeton.edu/#rgbd-reconstruction-datasets)

|                                         Name                                         | Depth Image | Color Image | Ground Truth Pose | Calibration |
| :----------------------------------------------------------------------------------: | :---------: | :---------: | :---------------: | :---------: |
|                        [SUN3D](http://sun3d.cs.princeton.edu/)                       |      X      |      X      |         X         |     Yes     |
| [7-Scenes](https://www.microsoft.com/en-us/research/project/rgb-d-dataset-7-scenes/) |      X      |      X      |         X         |      No     |
|   [RGB-D Scenes v2](http://rgbd-dataset.cs.washington.edu/dataset/rgbd-scenes-v2/)   |      X      |      X      |         X         |     Yes     |
|          [BundleFusion](http://graphics.stanford.edu/projects/bundlefusion/)         |      X      |      X      |         X         |      ?      |
|         [Analysis by Synthesis](http://graphics.stanford.edu/projects/reloc/)        |      X      |             |         X         |      ?      |

#### Dataset Format

Each scene is a folder containing one or more RGB-D video sequences. The folder contents are as follows:

* **camera-intrinsics.txt** - a text file with depth camera intrinsics (3x3 matrix in homogeneous coordinates)
* **seq-XX**
  * **frame-XXXXXX.color.png** - a 24-bit PNG RGB color image
  * **frame-XXXXXX.depth.png** - a 16-bit PNG depth image, aligned to its corresponding color image. Depth is saved in millimeters (mm). Invalid depth is set to 0.
  * **frame-XXXXXX.pose.txt** - a text file with the camera pose of the frame (camera-to-world, 4x4 matrix in homogeneous coordinates and in meters)


# Point Cloud Datasets

## Point Cloud Datasets

|   Name  | Point Cloud | Colored Point Cloud |    Ground Truth   |
| :-----: | :---------: | :-----------------: | :---------------: |
|   TUM   |             |                     |                   |
| Redwood |      X      |          X          | high-reso scanner |

### Redwood

* paper: Colored Point Cloud Registration Revisited, ICCV 2017
* website: <http://redwood-data.org/>


# LiDAR SLAM Datasets

### KITTI Autonomous Driving

* Website: <https://www.cvlibs.net/datasets/kitti/eval_odometry.php>
* Karlsruhe Institute of Technology (KIT), German, 2012
* Provides a variety of modalities (vision, LiDAR, IMU, RTK, etc.) and application domains (odometry, stereo, depth estimation, semantics, object detection, etc.)
* The ground truth in sequence 08 is not accurate at the beginning.&#x20;
* For odometry, the accuracy is evaluated by taking a sliding window of size 100, 200, ..., 800 frames, respectively, and computing the transformation difference between the first pose and the last pose in this window, and comparing it with the corresponding ground truth difference. The step size for the sliding window can be a fixed number such as 10 frames. The result is averaged over the entire trajectories and also different sliding window sizes.

### NCLT Dataset

* Website: <http://robots.engin.umich.edu/nclt/>
* The University of Michigan North Campus Long-Term Vision and LIDAR Dataset, 2016
* Mounted on a segway wheeled mobile robot.
* Modalities: Camera, LiDAR, IMU, RTK GPS, Wheel Odometry

### Newer College Dataset (NCD)

* Website: <https://ori-drs.github.io/newer-college-dataset/>
* Sensors mounted on a handheld device.
* Modalities: Ouster LiDAR and Realsense camera.&#x20;

### The Oxford RobotCar Dataset

* Website: <https://robotcar-dataset.robots.ox.ac.uk/>
* Oxford, 2014-2015
* Modalities: images, LiDAR, GPS

### The Oxford Radar RobotCar Dataset

* Website: <https://oxford-robotics-institute.github.io/radar-robotcar-dataset/>
* 280 km of driving around Oxford, UK
* Modalities: Navtech CTS350-X Millimetre-Wave FMCW radar, dual Velodyne HDL-32E LIDARs

### MulRan Dataset for Urban Place Recognition

* Website: <https://sites.google.com/view/mulran-pr/dataset>

### The Complex Urban Dataset

* Website: <https://sites.google.com/view/complex-urban-dataset>

### The HILTI SLAM Challenge Dataset 2022

* Website: <https://hilti-challenge.com/dataset-2022.html>

### The voxgraph Drone Dataset

* Website: <https://github.com/ethz-asl/voxgraph>


# Optimization




---

[Next Page](/llms-full.txt/1)

