D3.1 Draft plugins for multi-view registration and rectification and automated multi-view depth estimation

Deliverable D3.1 describes the first set of draft plugins for registration/rectification and disparity estimation. It comprises registration/rectification for trifocal sequences in one plugin and disparity estimation and post-filtering for stereo and trifocal sequences in a second plugin.

Currently these plugins are written and tested for Mamba FX on Windows since it has the same functionalities as Mistika but runs on different platforms. Implementation of the plugin already considered portability to Linux.

The draft plugins work on 8bit and 16bit RGB(A) images. For registration currently only spatial multiplexed trifocal (side-by-side stitching of left, centre and right images) is supported. Disparity estimation supports stereo and trifocal images in spatial multiplexed format as well as using multiple inputs for the plugin.

The registration algorithm entirely relies on detected feature points in the recorded sequences [1]. Dedicated calibration charts are not needed. To properly cover the geometry of a trifocal system, a large set of feature points (a few thousand points are optimal) is needed. That number of feature points cannot be extracted from a single image triple. Therefore the plugin first has to run through a representative part of the sequence in “analysis mode”.

Warping can be enabled already in the analysis phase. Horizontal lines can be drawn into the warped images to allow a good visual check of the rectification quality. In addition the estimated baseline factor provides a good hint on the estimation quality.

The rectified and time coherent images for the stereo case or trifocal case will be used to make a first automatic estimation of disparities. For the stereo case one pair (left and right) is estimated in both directions by the L-HRM algorithm [2]. In the trifocal case two pairs (left and centre, right and centre) are estimated independently in a parallel way also by the L-HRM.

Every pair is estimated in both directions which results in two disparity maps for stereo and four disparity maps for trifocal. Disparities that occur only in one direction are most of the time occlusions and are handled as outliers (pixels without disparity information). This process is known as a left to right consistency check.

In the stereo case occlusions can only be filled by the filtering while for the trifocal case the second camera pair can be used to achieve a merged disparity map. This merged map combines two reliable disparity maps into one.

Registration and rectification plugins are described in detail including all user interface parameters.

D6.1 Collection and Provision of Existing Test Material

3FLEX aims at developing advanced post-production tools for multiview content with a focus on trifocal and stereo material. To provide project partners with sufficient test material in an early phase of the project, sequences captured from the following European research projects were collected:

  • 3D4YOU, FP7 Project, No. 215075
  • 2020 3D Media, FP7 Project, No. 215475
  • MUSCADE, FP7 Project, No. 247010

In addition material from a joint project of Fraunhofer HHI, ARRI and Disney including two technical test shoots and a short film at the end of the project can be used. In particular the short film footage was captured without any special consideration of the technical feasibility and is therefore the most authentic of the entire collection.

Finally trifocal material shot by a student from HFF “Konrad Wolf” in Potsdam during his Diploma Thesis with a documentary format in mind completes the provided material.

D2.1: 3FLEX Functional and Technical Requirements

The general objective of the 3FLEX project is to develop an advanced 2D/3D postproduction workflow and the necessary tools to integrate multi-view and depth information into common postproduction tasks. The new workflow will enhance existing 2D and 3D production workflows through increased quality, flexibility and efficiency. This will be achieved through the development of various plugins and their integration into the commonly used postproduction platforms Mocha and Mistika (Mamba).

After analyzing current 2D and 3D production workflow this report describes the derived 3FLEX workflow and the interaction between the individual modules. After that, the host applications Mistika (Mamba) and Mocha are introduced and the most important workflows are described. Furthermore, the capabilities and limitations of the OpenFX API are analyzed to assess its suitability for the plugin development. The functional and technical requirements of the individual modules have been derived in order to fulfil the requirements of the workflow while respecting the technical aspects of the host applications and the plugin interface. Finally, the report discusses evaluation scenarios with potential datasets and suitable evaluation criteria for the overall workflow and the individual modules.

D7.1 Project website

This document describes the different aspects related to the design, development and launch of the 3FLEX web site as part of the overall dissemination tasks included in WP7, task 3.

The website is composed basically of a set of static pages containing the information about the project objectives, members of the consortium, etc. and two sections which will require continuous updates, news and outcomes, based in a blog engine to facilitate the post and search of the contents.

The website is publicly available from M3, January, 2014, at the URL: http://www.3flex-project.eu

All partners have reviewed the website. Partners are expected to regularly review and contribute in order to keep the site up to date.