T 1088/13 () of 21.8.2018

Summary of Facts and Submissions

I. The appeal is against the decision to refuse European patent application No. 05 780 538.4, published as international application WO 2006/016788 A1.

II. The patent application was refused by the examining division on the grounds that claims 1 of the main and first auxiliary requests were unclear (Article 84 EPC) and that the subject-matter of claim 1 of the second auxiliary request did not involve an inventive step (Article 56 EPC) in view of document:

D1: JP 2000 261808 A.

The third auxiliary request was not admitted into the proceedings (Rule 137(3) EPC) because the subject-matter of claim 1 prima facie did not involve an inventive step in view of D1 and the common general knowledge of the skilled person.

III. The applicants (appellants) appealed against this decision.

IV. The board issued a summons to oral proceedings and stated in an annex to the summons that clarity and inventive step would have to be discussed at the oral proceedings. It also sent the machine translation of D1 which it had used to assess it.

V. In response, with a letter dated 13 July 2018, the appellants submitted amended claims of a main, a first and a second auxiliary request.

VI. Oral proceedings were held before the board on 21 August 2018. The appellants requested that the decision under appeal be set aside and that a European patent be granted on the basis of the claims of the main request filed during the oral proceedings of 21 August 2018.

VII. The independent claims of the main request read as follows:

"1. A method of estimating a motion of a panorama image containing 360° omni-directional view information along an x-axis, the method comprising:

padding a padding region connected to the left side of a basic reference frame to be used for motion estimation of the panorama image using a right border region of the basic reference frame, padding a padding region connected to the right side of the basic reference frame using a left border region thereof, and making a reference frame by using the padded basic reference frame as the reference frame;

estimating a motion vector of a current data unit of the panorama image, using motion vectors of a plurality of previous data units adjacent to the current data unit;

determining a value of each pixel of a reference data unit indicated by the estimated motion vector from the reference frame (i) as the value of the pixel when the pixel belongs to the reference frame; and (ii) as the value of another pixel with an x coordinate determined by setting an x-coordinate of the pixel to a value obtained by adding a distance on the x-axis between a border of the basic reference frame adjacent to the pixel and the pixel to an x-coordinate of the opposite border of the basic reference frame when the pixel is located outside the right border of the reference frame, and (iii) as the value of another pixel with an x coordinate determined by setting an x-coordinate of the pixel to a value obtained by subtracting a distance on the x-axis between the border of the basic reference frame adjacent to the pixel and the pixel from the x-coordinate of the opposite border of the basic reference frame when the pixel is located outside the left border of the reference frame; and

determining a similarity between the current data unit and the reference data unit using a predetermined evaluation function,

wherein the motion vectors of the plurality of previous data units are detected before the motion vector of the current data unit is estimated."

"3. An apparatus for estimating a motion of a panorama image containing 360° omni-directional view information along an x-axis, the apparatus comprising:

a memory for storing a reference frame and motion vectors of a plurality of previous data units adjacent to a current data unit of the panorama image, the reference frame being obtained by padding a padding region connected to the left side of a basic reference frame to be used for motion estimation of the panorama image using a right border region of the basic reference frame, padding a padding region connected to the right side of the basic reference frame using a left border region thereof, and making a reference frame by using the padded basic reference frame as the reference frame; and

a motion estimating unit for estimating a motion vector of the current data unit using the motion vectors of the previous data units; determining a value of each pixel of the reference data unit indicated by the estimated motion vector from the reference frame (i) as the value of the pixel when the pixel indicated by the estimated motion vector belongs to the reference frame; (ii) as the value of another pixel with an x coordinate determined by setting an x-coordinate of the pixel to a value obtained by adding a distance on an x-axis between a border of the basic reference frame adjacent to the pixel and the pixel to an x-coordinate of the opposite border of the basic reference frame when the pixel is located outside the right border of the reference frame, and (iii) as the value of another pixel with an x coordinate determined by setting an x-coordinate of the pixel to a value obtained by subtracting a distance on the x-axis between the border of the basic reference frame adjacent to the pixel and the pixel from the x-coordinate of the opposite border of the basic reference frame when the pixel is located outside the left border of the reference frame; and determining a similarity between the current data unit and the reference data unit using a predetermined evaluation function,

wherein the motion vectors of the plurality of previous data units are detected before the motion vector of the current data unit is estimated."

"5. A method of compensating for a motion of a panorama image containing 360° omni-directional view information along an x-axis, the method comprising:

padding a padding region connected to the left side of a basic reference frame to be used for motion compensation for the panorama image using a right border region of the basic reference frame, padding a padding region connected to the right side of the basic reference frame using a left border region thereof, and making a reference frame by using the padded basic reference frame as the reference frame;

receiving a motion vector of a current data unit of the panorama image;

determining a value of each pixel of the reference data unit indicated by the received motion vector from the reference frame (i) as the value of the pixel of the reference data unit indicated by the motion vector of the current data unit when the pixel belongs to the reference frame; and (ii) as the value of another pixel with an x coordinate determined by setting an x-coordinate of the pixel to a value obtained by adding a distance on an x-axis between a border of the basic reference frame adjacent to the pixel and the pixel to an x-coordinate of the opposite border of the basic reference frame when the pixel is located outside the right border of the reference frame, and (iii) as the value of another pixel with an x coordinate determined by setting an x-coordinate of the pixel to a value obtained by subtracting the distance on the x-axis between the border of the basic reference frame adjacent to the pixel and the pixel from the x-coordinate of the opposite border of the basic reference frame when the pixel is located outside the left border of the reference frame; and reproducing the current data unit using the values of the pixels of the reference data unit.

6. An apparatus for compensating for a motion of a panorama image containing 360° omni-directional view information along an x-axis, the apparatus comprising:

a memory for storing a reference frame obtained by padding a padding region connected to the left side of a basic reference frame to be used for motion compensation for the panorama image using a right border region, padding a padding region connected to the right side of the basic reference frame using a left border region, and making a reference frame by using the padded basic reference frame as the reference frame; and

a motion compensating unit for receiving a motion vector of a current data unit of the panorama image; determining a value of each pixel of a reference data unit indicated by the received motion vector from the reference frame (i) as the value of the pixel when the pixel belongs to the reference frame; (ii) as the value of another pixel with an x coordinate determined by setting an x coordinate of the pixel to a value obtained by adding a distance on an x-axis between a border of the basic reference frame adjacent to the pixel and the pixel to an x-coordinate of the opposite border of the basic reference frame when the pixel is located outside the right border of the reference frame, and (iii) as the value of another pixel with an x coordinate determined by setting an x-coordinate of the pixel to a value obtained by subtracting the distance on the x-axis between the border of the basic reference frame adjacent to the pixel and the pixel from the x-coordinate of the opposite border of the basic reference frame when the pixel is located outside the left border of the reference frame; and reproducing the current data unit using the pixel values of the reference data unit.

7. A computer readable medium having stored instructions adapted to control a computer to perform a method of one of claims 1, 2 or 5."

Claims 2 and 4 are dependent on claims 1 and 3, respectively.

VIII. In the decision under appeal the examining division held that claims 1 of the then main and first auxiliary requests were unclear in so far as they referred to "determining values of all pixels ... by obtaining a value of a corresponding sub pixel". It was unclear how the values of all pixels could be determined by obtaining only one value of a sub pixel. It was also unclear whether claim 1 "aims at defining three alternatives of how to proceed in order to obtain a pixel value, including an unrestricted motion vector pointing to a region outside the reference frame, or whether the claimed subject-matter refers to only one case, without the possibility of having unrestricted motion vectors" (see decision under appeal, Reasons, sections 2.1 and 2.2).

D1 was considered the closest prior art for the subject-matter of claim 1 of the then second auxiliary request. D1 disclosed "taking into account the features of a panorama image, namely that the information of the left border is a continuation of the information of the right border and vice versa". The board understands the decision such that the examining division acknowledged that the steps detailing how the pixel values were determined using the panorama image constraint were not disclosed in D1. D1 also did not disclose determining a similarity between the current data unit and the reference data unit, but this was a common step in motion estimation. The technical problem associated with the determination of the pixel values was how to determine the values of pixels that formed a reference data unit in a panorama image. Starting from D1 the skilled person would "come to the idea of connecting regions from one border of a panorama image to the opposite border in order to improve the efficiency of motion estimation." The padding step did not "have an effect on the manner of treating pixels outside the basic reference frame, since the basic reference frame is formed by [the] original frame without the padding regions". The examining division also argued in the context of the then third auxiliary request that "either performing the padding in advance and storing a larger reference area or calculating the required pixel positions at the time those pixels are needed for motion compensation" was an obvious design choice (see decision under appeal, Reasons, section 2.3 and point 2.4.2).

Reasons for the Decision

1. The appeal is admissible.

2. The invention

The invention as claimed relates to a method and apparatus for estimating the motion of a panorama image containing 360º omnidirectional image information, and a method and apparatus for compensating for the motion of the panorama image.

In panorama images with 360º omnidirectional view, there is a high degree of resemblance between the image sections close to their left and right borders, since the panorama image ideally covers a cylindrical field of view. This property can be used for motion estimation and compensation in an unrestricted motion vector (UMV) mode, in which some or all pixels of a macro block that is to be coded may be predicted from a reference macro block that is (partially) located outside the image borders. According to the invention, border regions at the left and right sides of the panorama image (termed "basic reference frame" in claim 1) are padded to the panorama image's right and left borders, respectively, thereby creating an extended/padded basic reference frame (also termed "reference frame"). The pixel values in the reference frame are used to compute a measure of similarity between a current macro block (termed "current data unit") and a reference data unit.

The invention defines rules of how to access pixels of a reference data unit. If a pixel of the reference data unit is located in the reference frame, the pixel value is directly available in the reference frame. Otherwise the position of a corresponding pixel in the reference frame has to be computed, taking into account the cylindrical image properties of the panorama image with 360º omnidirectional view (see paragraphs [1], [2], [11], [51], [80] to [92]).

3. Amendments (Article 123(2) EPC)

3.1 The present set of claims differs from the claims of the main request underlying the decision under appeal in that it has been restricted to relate to pixels only, all references to sub-pixels having been deleted. In addition, claim 1 specifies that the panorama image contains 360º omni-directional view information "along an x-axis". The feature specifying how the pixel values are determined has been amended to read in its initial passage "determining a value of each pixel of a reference data unit ...". The part relating to calculation of the pixel value in the reference frame corresponding to the pixel in the reference data unit has been further rephrased to separate the conditions that the pixel is located outside the left or right borders of the reference frame.

3.2 The reference to (integer) pixels replacing those to sub-pixels is based on paragraphs [53], [85], [86], [102] and figures 14A and 14B of the application as published. A basis for the amendment specifying that the panorama image contains 360º omni-directional view information "along an x-axis" can be found in figures 4, 7, 13 and 15 together with original claims 1 and 4. The rephrasing of the determining feature is based on paragraphs [85] and [86] together with figures 14A and 14B.

3.3 The further claims 2 to 7 have been amended accordingly to specify the corresponding motion estimation apparatus, and a corresponding motion compensation method and apparatus as well as a computer readable medium having stored instructions adapted to control a computer to perform a method corresponding to the method claims. The feature concerning the detection of the motion vectors "of the previous data units" has been deleted from the claims referring to motion compensation.

3.4 Hence, the board finds that the claims of the appellants' sole request do not contain subject-matter which extends beyond the content of the application as filed and that they thus comply with Article 123(2) EPC.

4. Clarity (Article 84 EPC 1973)

4.1 The claims have been revised to overcome the objections in the decision under appeal (see point VIII above). In particular, the reference to "sub pixels" and a "corresponding sub pixel" has been replaced and the determining feature has been rephrased to clearly specify three options for determining pixel values of a reference data unit. These options have been itemised as options (i) to (iii).

4.2 The board understands the reference to "previous data units" in claim 1 to refer to data units / macroblocks in the same frame as the current data unit and for which the motion vector has been estimated previously, i.e. before the motion vector of the current data unit is determined. In accordance with claim 1, motion vectors from these data units which are adjacent to the current data unit are used to derive an estimate of the current data unit's motion vector, e.g. by using the median of the motion vectors of adjacent blocks. The board is of the opinion that the skilled person would understand the reference in claim 1 to "previous data units" in that sense, especially since this is a conventional method of estimating a motion vector and is described as such in the application (see paragraphs [6] to [9]).

4.3 The board can see no other expressions in claim 1 which could raise doubts as to clarity. Moreover, the further claims 2 to 7 have been revised to provide clarifications similar to those for claim 1.

4.4 Hence, the board finds the present set of claims to comply with Article 84 EPC 1973.

5. Inventive step (Article 56 EPC 1973)

5.1 It is common ground that D1 may be considered the closest prior art for the present application. In this section of the decision, all references to specific paragraphs of D1 refer to the machine translation (see point IV).

5.2 D1 discloses a method of encoding a panorama image containing 360º omnidirectional view information along an x-axis including estimation of motion vectors (see abstract).

5.3 However, D1 fails to disclose the remaining steps of claim 1. In particular, D1 does not disclose the following features (which have been put in a slightly different order from in claim 1):

(a) padding a padding region connected to the left side of a basic reference frame to be used for motion estimation of the panorama image using a right border region of the basic reference frame, padding a padding region connected to the right side of the basic reference frame using a left border region thereof, and making a reference frame by using the padded basic reference frame as the reference frame;

(b) estimating a motion vector of a current data unit of the panorama image, using motion vectors of a plurality of previous data units adjacent to the current data unit;

... wherein the motion vectors of the plurality of previous data units are detected before the motion vector of the current data unit is estimated;

(c) determining a value of each pixel of a reference data unit indicated by the estimated motion vector from the reference frame (i) as the value of the pixel when the pixel belongs to the reference frame; and (ii) as the value of another pixel with an x-coordinate determined by setting an x-coordinate of the pixel to a value obtained by adding a distance on the x-axis between a border of the basic reference frame adjacent to the pixel and the pixel to an x-coordinate of the opposite border of the basic reference frame when the pixel is located outside the right border of the reference frame, and (iii) as the value of another pixel with an x-coordinate determined by setting an x-coordinate of the pixel to a value obtained by subtracting a distance on the x-axis between the border of the basic reference frame adjacent to the pixel and the pixel from the x-coordinate of the opposite border of the basic reference frame when the pixel is located outside the left border of the reference frame,

(d) determining a similarity between the current data unit and the reference data unit using a predetermined evaluation function.

5.4 On feature (a) the examining division argued in the decision under appeal that D1's figure 8 showed blocks located at the left border being padded and connected to the right border in order to perform motion estimation based on an area formed by adjacent units (see Reasons, points 2.3.2 and 2.3.7).

The board agrees with the decision under appeal that D1 discloses using the cylindrical image property of panorama images containing 360º omnidirectional view to estimate motion vectors (see figure 8 and paragraphs [0062], [0063]). Hence, D1 discloses retrieving pixel values of pixels outside the right border of the image (corresponding to the basic reference frame of claim 1) from corresponding pixels close to the left border of the image. However, D1 does not disclose "making a reference frame" by using the "padded basic reference frame".

5.5 The novelty of feature (b) was not discussed in the decision under appeal, because it and, similarly, feature (d) were described in the application as relating to a conventional motion estimation technique and therefore not decisive for inventive step (see paragraphs [6] to [10] of the application). Nevertheless, the board finds that neither of features (b) and (d) is disclosed in D1.

5.6 On the determining step (feature (c)) the examining division argued that "pixels outside the basic reference frame are treated in the same manner, irrespective of whether they are in the padded region or outside the reference frame" (see decision under appeal, Reasons, point 2.3.7). This argument does not apply to the present set of claims at least, because pixels in the basic reference frame and pixels in the padded regions (outside the basic reference frame, but inside the reference frame) are processed according to condition (i) whereas pixels outside the padded regions are subject to conditions (ii) or (iii). Hence, feature (c) is not disclosed in D1.

5.7 For the above reasons (see point 5.5), the discussion on inventive step focused on the effects provided by features (a) and (c).

5.8 The appellants argued that features (a) and (c) allowed fast motion estimation (and compensation) due to the fact that pixel values in the reference frame, i.e. the basic reference frame with padding regions, could be directly retrieved without additional coordinate calculations (see letter dated 13 July 2018, page 3, penultimate paragraph).

5.9 The board agrees and considers the corresponding technical problem to be how to improve the speed of motion estimation and compensation in the case of panorama images with 360º omnidirectional view.

5.10 The examining division argued in this respect that "either performing padding in advance and storing a larger reference area or calculating the required pixel positions at the time those pixels are needed for motion compensation" was an obvious design choice (see point VIII above).

5.11 The examining division argued correctly that D1 discloses using the cylindrical image property of panorama images with 360º omnidirectional view. However, this does not imply performing a padding operation for fast retrieval of pixel values. Instead, pixel values for pixels outside the borders of the image can be retrieved by calculating the corresponding pixel positions in the basic reference frame. In addition, claim 1 specifies a combination of both options, namely retrieval of the value of the (target) pixel indicated by the estimation motion vector using a padded reference frame (step (i)) and processing by using calculations to determine the position of the (target) pixel (steps (ii) and (iii)), depending on whether or not the (target) pixel belongs to the padded reference frame.

5.12 The further documents cited in the proceedings before the examining division also do not disclose or suggest the determining steps.

5.13 Hence, the board holds that the subject-matter of claim 1 involves an inventive step over the cited prior art.

5.14 The same assessment applies for the corresponding claims 3, 5 and 6, which specify the corresponding apparatus for estimating the motion of a panorama image and a method and apparatus for compensating for the motion of the panorama image. Claims 2 and 4 are dependent claims, and claim 7 specifies the corresponding computer readable medium.

6. Remittal to the first instance (Article 111(1) EPC 1973)

The claims according to the sole request meet the requirements of the EPC. The description, however, has not yet been adapted. In view of the complexity of the application, which comprises multiple embodiments, and the fact that an allowable set of claims was only filed during the oral proceedings, the board exercises its discretion pursuant to Article 111(1) EPC 1973 to remit the case to the department of first instance to deal with the adaptation of the description.

Order

For these reasons it is decided that:

1. The decision under appeal is set aside.

2. The case is remitted to the department of first instance with the order to grant a patent with the following claims and a description to be adapted thereto:

Claims No. 1 - 7 of the main request filed during the oral proceedings of 21 August 2018.