Commit de4a57ed authored by Alejandro Homs Puron's avatar Alejandro Homs Puron
Browse files

Merge branch 'geometry-transformations' into 'develop'

Geometry transformations

See merge request !27
parents 86edd09c aa4e505b
Pipeline #25267 failed with stages
in 74 minutes and 44 seconds
......@@ -62,6 +62,7 @@ include(options.cmake)
find_package(Boost COMPONENTS
program_options
unit_test_framework
filesystem
system
REQUIRED
)
......
......@@ -79,53 +79,104 @@ TODO
## Detector modules layout
A detector layout is composed of modular items. Each item can be rotated and/or
flipped due to detector readout electronics design. In such cases, the isometric
transformation necessary to reconstruct the data in the final layout must be
defined. See :doc:`geometry_transformations`.
Further operations can be performed on the full layout, affecting all the
items. One specific operation is image crop (RoI), which may result in the
exclusion of one or more items from the layout. Those "not-in-layout" items are
nevertheless referenced by the layout structure in order to not change the
item index list.
The layout is just a description of how the modular items are arranged.
In order to assemble the final image the source data must be provided. The
implementation expects one source view per layout item, even "not-in-layout"
items.
An affine matrix transforming from sensor coordinates to final image coordinates
is kept by the layout. This is useful when the user wants to keep the same
region-of-interest in the sensor while adding/removing geometric transformations.
``` c++
struct layout_item
{
enum rotation {
rotated_0,
rotated_90cw,
rotated_90ccw,
rotated_180
};
// Position in the destination
point_t topleft;
rotation rotated;
point_t dst_topleft;
// Selection within the source
point_t select_topleft;
point_t select_dimensions;
point_t src_topleft;
point_t src_dimensions;
// Transformation to be applied on item to reconstruct the layout
any_isometric_xform_t xform; // default is none
bool in_layout() const;
image_rect get_src_bounding_box() const;
image_rect get_dst_bounding_box() const;
point_t get_dst_dimensions() const;
};
// A layout is basically a container of items
struct layout
{
typedef point2<std::ptrdiff_t> point_t;
// Add a layout item
// Add/retreive a layout item
void add_item(layout_item i);
const layout_item& get_item(int i) const;
std::size_t size() const;
// Return/set the dimension of the full image
const point_t& get_dims() const;
void set_dims(point_t dims);
// converts sensor pixel coordinates into layout coordinates
affine_t get_sensor_coordinates_matrix() const;
// Returns the dimension of the full image
const point_t& dim();
...
};
```
### Reconstruction
### Transformation
```cpp
// Reconstruction to memory
template <typename Range, typename Image, typename Value>
void assemble(const Range& views, Image& out, Value fillvalue) const;
The layout implements the following transformations:
// Reconstruction to file
template <typename Range, typename T>
void generate_vds(const std::string& filename, Range h5_sources, T fillvalue) const;
```c++
// Image algorithms
void vert_flip();
void horz_flip();
void rotate90cw();
void rotate90ccw();
void rotate180();
void crop(point_t topleft, point_t dimensions);
```
### Transformation
Each transformation updates the item state as well as the layout dimensions
and the sensor-coordinates-matrix.
*TODO*.
See :doc:`geometry_transformations`.
### Reconstruction
The layout reconstruction can be performed in two variants
* Memory. This implementation uses serial Boost.Gil algorithms to reconstruct
the layout items into a single image:
``` c++
// Reconstruction to memory
template <typename Range, typename Image, typename Value>
void assemble(const Range& views, Image& out, Value fillvalue) const;
```
* Virtual data set: TODO
``` c++
// Reconstruction to file
template <typename Range, typename T>
void generate_vds(const std::string& filename, Range h5_sources, T fillvalue) const;
```
## Usage
......
# Geometric Transformations
Several geometric image transformations are implemented in Lima:
- pixel binning,
- crop (also known as Region-of-Interest: RoI),
- vertical (top-down) / horizontal (left-right) flip and
- rotation by 90 degrees, both clockwise (CW) and counter-clockwise (CCW), and 180 degrees.
These operations are normally applied on the generated images on user demand. Several detectors can implement pixel binning and or RoIs, typically increasing the effective frame rate or reducing the data bandwidth. Other operations like flip do not affect the detector timing but avoids the need of extra software processing, so hardware-acceleration is preferred.
Isometric transformations like flip and rotation are also necessary to reconstruct the raw images sent by tiled detectors, in which constrains in the detector head design impose different readout sequences for different modules. Lima follows the default image convention where column index X increases towards the right direction and row index Y increases towards the bottom, so <0,0> is the top-left corner. The corresponding flip and/or rotation must be performed before assembling each elemental region in order to follow this rule.
The `lima::processing::geom` namespace provides helpers to manipulate the coordinates during the application of geometric transformations.
## Affine Transformation Matrix
Each geometric transformation can be described by an affine Boost.Gil 3x2 matrix (actually a 3x3 matrix whose last column is always <0,0,1>). The matrix converts the coordinates of a pixel at the input of the transformation to its corresponding pixel in the output:
```math
\begin{bmatrix} x' \\ y' \\ 1 \end{bmatrix} = \begin{bmatrix} a & b & 0 \\ c & d & 0 \\ d & e & 1 \end{bmatrix} \begin{bmatrix} x \\ y \\ 1 \end{bmatrix}
```
The third dimension allows translations in the coordinate system.
For scaling transformations, such as binning, the arithmetic precision can be guarantied with rational number arithmetic (e.g. Boost.Rational) or arbitrary precision arithmetic (e.g. Boost.Multiprecision):
```c++
using integral_t = std::ptrdiff_t;
using point_t = boost::gil::point<integral_t>;
using rational_t = boost::rational<integral_t>;
using affine_t = boost::gil::matrix3x2<rational_t>;
```
The affine matrix associated to a sequence of transformation is the multiplication of the individual matrices:
```math
A = A_1 . A_2 ... A_n
```
The `boost::gil::inverse` function returns a transformation matrix to input coordinates from output ones:
```c++
point_t output = input * xform_affine;
BOOST_CHECK(input == output * boost::gil::inverse(xform_affine));
```
To get the coordinate of a pixel after a given transformation, a number of step
- Transform the pixel coordinate to geometric coordinate (e.g. offset to the center of the pixel p' = p + {0.5, 0.5})
- Move the image to the center of the coordinate system
- Apply the transformation matrix m (p' = m * p)
- Move the image back to the first quadrant of the coordinate system
- Transform the geometric coordinate to pixel coordinate (e.g. offset to the center of the pixel p' = p - {0.5, 0.5})
## Available basic transformations
* `bin`
* `crop`
* `flip_up_down`
* `flip_left_right`
* `rotate_90ccw`
* `rotate_180`
* `rotate_90cw`
## Isometric transformations
Eight isometric transformations are identified in Lima:
* isometric_xform_none
* isometric_xform_vflip
* isometric_xform_hflip
* isometric_xform_rot90ccw
* isometric_xform_rot180
* isometric_xform_rot90cw
* isometric_xform_vflip_rot90ccw
* isometric_xform_hflip_rot90ccw
The *any_isometric_xform_t* is a *std::variant* of these transformations.
The implementation represents an isometric transformation as a combination of
three basic operations: vertical flip, horizontal flip and rotation 90 CCW. In
particular, it is a Boost.Hana tuple of three booleans:
```c++
#define LIMA_ISO_XFORM_TUPLE(v, h, r) \
boost::hana::tuple<boost::hana::bool_<v>, \
boost::hana::bool_<h>, \
boost::hana::bool_<r>>
```
The result of the successive application of two transformations, which is
also an isometric transformation, is given by the *operator +*. It is implemented
to provide a constexpr, as well as its inverse *operator -*. For any given
transformation, its *inverse* can always obtained:
```c++
template <bool V, bool H, bool R>
constexpr auto inverse(const isometric_xform_base_t<V, H, R>& t)
{
return isometric_xform_none() - t;
}
```
## Helpers
The following helpers are also defined in `lima::processing::geometry`:
* `rectangle`: represents a selection in a image, providing inclusion and
overlapping calculations
* get_affine: function template returning the affine matrix of an arbitrary
transformation
* get_output_dimensions: function template returning the output dimensions
after an arbitrary transformation
* apply_isometric_xform: result of an rectangle after a transformation
......@@ -67,6 +67,7 @@ Note that this documentation is also available in pdf and epub format.
:caption: Processing
processing
geometric_transformations
.. toctree::
:maxdepth: 2
......
......@@ -38,7 +38,7 @@ where `pipeline.json` is the pipeline definition:
## Library
This short tutprila should get you started with the processing library of Lima2. First some includes are required:
This short tutorial should get you started with the processing library of Lima2. First some includes are required:
``` c++
#include <lima/processing/pipeline.hpp>
......
......@@ -7,6 +7,7 @@
#pragma once
#if !defined(LIMA_LAYOUT_HPP)
#define LIMA_LAYOUT_HPP
#include <algorithm>
#include <iostream>
......@@ -15,31 +16,46 @@
#include <boost/gil.hpp>
#include <boost/range/combine.hpp>
namespace lima {
namespace gil = boost::gil;
#include <lima/processing/geom/rectangle.hpp>
#include <lima/processing/geom/isometric.hpp>
template <typename T>
using point2 = boost::gil::point2<T>;
namespace lima {
struct layout_item
{
typedef point2<std::ptrdiff_t> point_t;
enum rotation {
rotated_0,
rotated_90cw,
rotated_90ccw,
rotated_180
};
using point_t = boost::gil::point<std::ptrdiff_t>;
using rectangle_t = processing::geom::rectangle<std::ptrdiff_t>;
using any_isometric_xform_t = processing::geom::any_isometric_xform_t;
// Position in the destination
point_t topleft;
rotation rotated;
point_t dst_topleft;
// Selection within the source
point_t select_topleft;
point_t select_dimensions;
point_t src_topleft;
point_t src_dimensions;
// Transformation to be applied on item to reconstruct the layout
any_isometric_xform_t xform; // default is none
bool in_layout() const
{
return (src_dimensions.x > 0) && (src_dimensions.y > 0);
}
rectangle_t get_src_bounding_box() const
{
return {src_topleft, src_dimensions};
}
rectangle_t get_dst_bounding_box() const
{
return {dst_topleft, get_dst_dimensions()};
}
point_t get_dst_dimensions() const
{
return processing::geom::output_dimensions(xform, src_dimensions);
}
};
......@@ -53,52 +69,70 @@ struct layout_item
class layout
{
public:
typedef point2<std::ptrdiff_t> point_t;
void add_item(layout_item i)
{
// Check if sources do not overlap
// Update dimensions using bounding box
m_dimensions.x = std::max(m_dimensions.x, i.topleft.x + i.select_dimensions.x);
m_dimensions.y = std::max(m_dimensions.y, i.topleft.y + i.select_dimensions.y);
using point_t = layout_item::point_t;
using rectangle_t = layout_item::rectangle_t;
using affine_t = processing::geom::affine_t;
m_items.push_back(i);
}
void add_item(layout_item i);
const point_t& dim() const { return m_dimensions; }
void dim(const point_t& dim) { m_dimensions = dim; }
const layout_item& get_item(int i) const { return m_items[i]; }
std::size_t size() const { return m_items.size(); }
point_t get_dims() const { return m_dimensions; }
void set_dims(point_t dims) { m_dimensions = dims; }
//// Basic algorithms
//template <typename UnaryFunction >
//UnaryFunction for_each_item(UnaryFunction func) const { std::for_each(m_items.begin(), m_items.end(), func); return func; }
// Image algorithms
void flip();
void vert_flip();
void horz_flip();
void rotate90cw();
void rotate90ccw();
void rotate180();
void crop(const point_t& topleft, const point_t& dimensions);
void crop(rectangle_t roi);
// Reconstruction
template <typename Range, typename Image, typename Value>
void assemble(const Range& views, Image& out, Value fillvalue) const;
template <typename Range, typename T>
void generate_vds(const std::string& filename, Range h5_sources, T fillvalue) const;
void generate_vds(const std::string& filename, Range h5_sources,
T fillvalue) const;
// converts sensor pixel coordinates into layout coordinates
affine_t get_sensor_coordinates_matrix() const
{ return m_sensor_coordinates_matrix; }
private:
template <bool V, typename Image = int>
static auto xform_vflip(const Image& src);
template <bool H, typename Image = int>
static auto xform_hflip(const Image& src);
template <bool R, typename Image = int>
static auto xform_rot90(const Image& src);
template <typename Image, bool V, bool H, bool R>
static auto xform_view(const Image& src,
const processing::geom::isometric_xform_base_t<V, H, R>& t);
template <typename T>
void apply_xform(const T& t);
point_t m_dimensions;
std::vector<layout_item> m_items;
affine_t m_sensor_coordinates_matrix;
};
// A memory source for the layout
template <typename Image>
struct mem_source
{
typedef Image image_t;
using image_t = Image;
mem_source(const image_t& source) : m_source(source) { }
......@@ -120,6 +154,28 @@ struct h5_source
std::string m_dataset;
};
// Add a new item to the layout
void layout::add_item(layout_item i)
{
auto bounding_box = i.get_dst_bounding_box();
// Check if sources do not overlap
for (auto const& o: m_items) {
if (!o.in_layout())
continue;
auto other_bounding_box = o.get_dst_bounding_box();
if (overlaps(bounding_box, other_bounding_box))
throw std::invalid_argument("Layout items overlap");
}
// Update dimensions using bounding box
auto br = bounding_box.br_corner();
m_dimensions.x = std::max(m_dimensions.x, br.x + 1);
m_dimensions.y = std::max(m_dimensions.y, br.y + 1);
m_items.push_back(i);
}
// Generate an HDF5 virtual dataset
template <typename Range, typename T>
void layout::generate_vds(const std::string& filename, Range h5_sources, T fillvalue) const
......@@ -127,6 +183,40 @@ void layout::generate_vds(const std::string& filename, Range h5_sources, T fillv
// TODO
}
template <bool V, typename Image>
auto layout::xform_vflip(const Image& src)
{
if constexpr (V)
return boost::gil::flipped_up_down_view(src);
else
return src;
}
template <bool H, typename Image>
auto layout::xform_hflip(const Image& src)
{
if constexpr (H)
return boost::gil::flipped_left_right_view(src);
else
return src;
}
template <bool R, typename Image>
auto layout::xform_rot90(const Image& src)
{
if constexpr (R)
return boost::gil::rotated90ccw_view(src);
else
return src;
}
template <typename Image, bool V, bool H, bool R>
auto layout::xform_view(const Image& src,
const processing::geom::isometric_xform_base_t<V, H, R>& t)
{
return xform_rot90<R>(xform_hflip<H>(xform_vflip<V>(src)));
}
// Assemble the final image from the layout
template <typename Range, typename Image, typename Value>
void layout::assemble(const Range& views, Image& out, Value fillvalue) const
......@@ -137,7 +227,7 @@ void layout::assemble(const Range& views, Image& out, Value fillvalue) const
// Resize output according to layout dimension
out.recreate(m_dimensions);
gil::fill_pixels(gil::view(out), fillvalue);
boost::gil::fill_pixels(boost::gil::view(out), fillvalue);
//for (auto [item, view]: boost::combine(m_items, views))
for (auto const& i: boost::combine(m_items, views))
......@@ -149,56 +239,106 @@ void layout::assemble(const Range& views, Image& out, Value fillvalue) const
boost::tie(item, view) = i;
//}
// Copy to the destination line by line
boost::gil::copy_pixels(
gil::subimage_view(view, item.select_topleft, item.select_dimensions),
gil::subimage_view(gil::view(out), item.topleft, item.select_dimensions)
);
if (!item.in_layout())
continue;
// Copy to the destination line by line, applying xforms
auto src_view = boost::gil::subimage_view(view,
item.src_topleft,
item.src_dimensions);
auto dst_view = boost::gil::subimage_view(boost::gil::view(out),
item.dst_topleft,
item.get_dst_dimensions());
std::visit([&](auto&& t) {
boost::gil::copy_pixels(xform_view(src_view, t), dst_view);
}, item.xform);
}
}
/// Apply transformation to item
template <typename Xform>
void layout::apply_xform(const Xform& t)
{
// Transform each item
for (auto& item: m_items) {
if (!item.in_layout())
continue;
rectangle_t bb0 = item.get_dst_bounding_box();
rectangle_t bb1 = processing::geom::apply_isometric_xform(m_dimensions, bb0, t);
item.dst_topleft = bb1.topleft;
std::visit([&](auto const& x) {
//item.xform = x + t;
item.xform = processing::geom::operator+(x, t);
}, item.xform);
}
// sensor coordinates
affine_t& m = m_sensor_coordinates_matrix;
m = m * boost::gil::complete_affine(processing::geom::affine(t), m_dimensions);
// dimensions
m_dimensions = processing::geom::output_dimensions(t, m_dimensions);
}
/// Flip layout
inline void layout::flip()
inline void layout::vert_flip()
{
apply_xform(processing::geom::isometric_xform_vflip());
}
inline void layout::horz_flip()
{
apply_xform(processing::geom::isometric_xform_hflip());
}
/// Rotate layout 90 degree clockwise
inline void layout::rotate90cw()
{
// Swap layout dimension
std::swap(m_dimensions.x, m_dimensions.y);
for (auto& item: m_items)
{
// Swap every items dimension
std::swap(item.select_dimensions.x, item.select_dimensions.y);
// Mark item as rotated
// Rotate
std::swap(item.topleft.x, item.topleft.y);
item.topleft.x = m_dimensions.x - item.topleft.x - item.select_dimensions.x;
}
apply_xform(processing::geom::isometric_xform_rot90cw());
}
/// Rotate layout 90 degree counterclockwise
inline void layout::rotate90ccw()
{
// Swap layout dimension
std::swap(m_dimensions.x, m_dimensions.y);
apply_xform(processing::geom::isometric_xform_rot90ccw());
}
/// Rotate layout counterclockwise 180 degree
inline void layout::rotate180()
{
apply_xform(processing::geom::isometric_xform_rot180());
}
/// Crop the layout
inline void layout::crop(const point_t& topleft, const point_t& dimensions)
inline void layout::crop(rectangle_t rec)
{