T 1826/15 () of 18.8.2020

Summary of Facts and Submissions

I. The appeal is against the decision to refuse European patent application No. 07 808 525.5, published as international application WO 2008/016305 A2.

II. The decision under appeal cited the following documents:

D1:|JP 2005 274142 A.|

D2:|TOET, Alexander. Natural colour mapping for multiband nightvision imagery. Information Fusion, volume 4, issue 3, September 2003, pages 155-156, XP003023442.|

D3:|GB 2 287 600 A.|

D4:|EP 0 319 684 A.|

D6:|US 2005/195453 A1. |

III. It was based on the ground that the subject-matter of the claims of the three requests then on file lacked inventive step within the meaning of Article 56 EPC.

IV. The applicant ("appellant") filed notice of appeal. With the statement of grounds of appeal, it requested that the decision under appeal be set aside and that a patent be granted on the basis of the claims of one of the four requests filed with the statement of grounds of appeal. The claims of the main request, first auxiliary request and third auxiliary request filed with the statement of grounds of appeal correspond to the claims of the main request, first auxiliary request and second auxiliary request forming the basis of the impugned decision, respectively. The appellant provided a basis for the amendments as well as arguments as to why the subject-matter of the claims of the requests involved an inventive step within the meaning of Article 56 EPC.

V. The board issued a summons to oral proceedings. In a communication under Article 15(1) RPBA 2020 (Rules of Procedure of the Boards of Appeal in the version of 2020, OJ EPO 2019, A63), the board gave its preliminary opinion that:

(a) the subject-matter of claims 1, 8 and 14 of the main request and of the first auxiliary request, the subject-matter of claims 1, 7 and 13 of the second auxiliary request, and the subject-matter of claims 1, 5 and 9 of the third auxiliary request lacked inventive step in view of the disclosure of document D2 combined with the common general knowledge of a person skilled in the art;

(b) the subject-matter of claim 15 of the main request and of the first auxiliary request and the subject-matter of claim 14 of the second auxiliary request was not novel over the disclosure of document D2;

(c) the subject-matter of claim 10 of the third auxiliary request lacked inventive step in view of the common general knowledge of a person skilled in the art.

VI. By letter dated 24 June 2020, the appellant informed the board that it would not be attending the oral proceedings. It did not comment on the board's preliminary opinion given in the board's communication. Subsequently, the oral proceedings were cancelled.

VII. Claim 1 of the main request reads as follows:

"Method for converting at least one image of a first spectrum into an image of a second spectrum, comprising:

- recording at least one first-spectrum multi-band reference image (NRI) of at least one reference scene (RS) with a first-spectrum recording apparatus (1), the first-spectrum multi-band reference image (NRI) comprising first image portions with corresponding first-spectrum sensor reference data (b1, b2), wherein the first-spectrum sensor reference data comprises multi-band data, relating to at least two different parts of the spectrum;

- providing corresponding second-spectrum reference information (RGB);

- providing at least one set of reference data (T1, T2) from at least part of the first-spectrum sensor reference data (b1, b2) relating to the reference scene (RS), and at least a corresponding part of the second-spectrum information (RGB) relating to that reference scene (RS);

- providing at least one target first-spectrum image (NTI) or target second-spectrum image; and

- converting each target first-spectrum image (NTI) or target second-spectrum image into a respective second-spectrum image (CI) or first-spectrum image, respectively, using the set of reference data (T1, T2), wherein the recording apparatus is a night-vision apparatus.".

VIII. Claim 1 of the first auxiliary request reads as follows (amendments to claim 1 of the main request are underlined):

"Method for converting at least one image of a first spectrum including infra-red wavelengths into an image of a second spectrum, the second spectrum being a broad visible spectrum, the method comprising:

- recording at least one first-spectrum multi-band reference image (NRI) of at least one reference scene (RS) with a first-spectrum recording apparatus (1), the first-spectrum multi-band reference image (NRI) comprising first image portions with corresponding first-spectrum sensor reference data (b1, b2), wherein the first-spectrum sensor reference data comprises multi-band data, relating to at least two different parts of the spectrum;

- providing corresponding second-spectrum reference information (RGB);

- providing at least one set of reference data (T1, T2) from at least part of the first-spectrum sensor reference data (b1, b2) relating to the reference scene (RS), and at least a corresponding part of the second-spectrum information (RGB) relating to that reference scene (RS);

- providing at least one target first-spectrum image (NTI) or target second-spectrum image; and

- converting each target first-spectrum image (NTI) or target second-spectrum image into a respective second-spectrum image (CI) or first-spectrum image, respectively, using the set of reference data (T1, T2), wherein the recording apparatus is a night-vision apparatus.".

IX. Claim 1 of the second auxiliary request reads as follows (amendments to claim 1 of the main request are underlined):

"Method for converting at least one image of a first spectrum into an image of a second spectrum, comprising:

- recording at least one first-spectrum multi-band reference image (NRI) of at least one reference scene (RS) with a first-spectrum recording apparatus (1), the first-spectrum multi-band reference image (NRI) comprising first image portions with corresponding first-spectrum sensor reference data (b1, b2), wherein the first-spectrum sensor reference data comprises multi-band data, relating to at least two different parts of the spectrum;

- providing corresponding second-spectrum reference information (RGB), wherein the second-spectrum reference information is true-colour colour information;

- providing at least one set of reference data (T1, T2) from at least part of the first-spectrum sensor reference data (b1, b2) relating to the reference scene (RS), and at least a corresponding part of the second-spectrum information (RGB) relating to that reference scene (RS);

- providing at least one target first-spectrum image (NTI) or target second-spectrum image; and

- converting each target first-spectrum image (NTI) or target second-spectrum image into a respective second-spectrum image (CI) or first-spectrum image, respectively, using the set of reference data (T1, T2), wherein the recording apparatus is a night-vision apparatus.".

X. Claim 1 of the third auxiliary request reads as follows (amendments to claim 1 of the main request are underlined):

"Method for converting at least one image of a first spectrum including infra-red wavelengths into an image of a second spectrum, the second spectrum being a broad visible spectrum, the method comprising:

- recording at least one first-spectrum multi-band reference image (NRI) of at least one reference scene (RS) with a first-spectrum recording apparatus (1), the first-spectrum multi-band reference image (NRI) comprising first image portions with corresponding first-spectrum sensor reference data (b1, b2), wherein the first-spectrum sensor reference data comprises multi-band data, relating to at least two different parts of the spectrum;

- providing corresponding second-spectrum reference information (RGB), the second-spectrum reference information being true-colour colour information;

- providing at least one set of reference data (T1, T2) from at least part of the first-spectrum sensor reference data (b1, b2) relating to the reference scene (RS), and at least a corresponding part of the second-spectrum information (RGB) relating to that reference scene (RS);

- providing at least one target first-spectrum image (NTI) or target second-spectrum image; and

- converting each target first-spectrum image (NTI) or target second-spectrum image into a respective second-spectrum image (CI) or first-spectrum image, respectively, using the set of reference data (T1, T2), wherein the recording apparatus is a night-vision apparatus, wherein the second-spectrum reference information (RGB) is already known, for example via visual inspection of a reference scene, or is being provided by providing at least one second-spectrum reference image (CRI) of each reference scene, the at least one second-spectrum reference image (CRI) comprising second, second-spectrum, image portions with corresponding second-spectrum information (RGB); wherein the set of reference data is being generated by: - selecting a plurality of first, first-spectrum, image portions and corresponding second, second-spectrum, image portions, such that reference scene image coordinates of a first image portion substantially correspond to reference scene image coordinates of the corresponding second image portion;- indexing the first-spectrum sensor reference data in a multi-band lookup table (T1) and indexing the second-spectrum information im (read: in) a natural colour lookup table (T2), such that each entry in a resulting index refers to the first-spectrum sensor reference data and the corresponding second-spectrum information associated with a selected first image portion and a corresponding second image portion;

the method, comprising, in case of conversion of target first-spectrum images (NTI) into respective second-spectrum images (CI):

- finding a closest match between first-spectrum sensor data associated with each target first-spectrum image portion of a target first-spectrum image (NTI) and the first-spectrum sensor reference data of the first image portions;

- replacing each first-spectrum target image portion with the second, second-spectrum, image portion, which second image portion is associated with the first image portion that most closely matched that target image portion, the matching being based on the first-spectrum sensor data, to generate the second-spectrum image (CI);

or, vice-versa, in case of conversion of target second-spectrum images into respective first-spectrum images:

- finding a closest match between second spectrum information associated with each target second-spectrum image portion of a target second-spectrum image and the second-spectrum information of the second image portions;

- replacing each second-spectrum target image portion with the first, first-spectrum, image portion, which first image portion is associated with the first image portion that most closely matched that target image portion, the matching being based on the second-spectrum information, to generate the first-spectrum image.".

XI. In its statement of grounds of appeal, the appellant essentially argued that a person skilled in the art would not combine the disclosure of document D2 with the disclosure of any of documents D1, D3, D4 and D6 because these documents did not concern the same problem as document D2. Even if a person skilled in the art were to combine the disclosure of document D2 with the disclosure of any one of these documents, the combination would not result in the claimed invention.

Reasons for the Decision

1. The appeal is admissible.

2. The invention

2.1 The present application relates to the conversion of a multi-band night-time image into an image with natural daytime colour appearance.

2.2 The method essentially amounts to building a lookup table associating night-time pixel values with natural daytime colours. This table is built by capturing reference images (for example of a same reference scene) in the daytime and night-time and by extracting values of corresponding regions of these reference images. Each pair of values is associated with an index in the lookup table. The table is then used to convert any night-time image into an image with natural daytime colour appearance, and vice versa.

3. Main request: inventive step (Article 56 EPC 1973)

3.1 According to Article 56 EPC 1973, an invention is to be considered as involving an inventive step if, having regard to the state of the art, it is not obvious to a person skilled in the art.

3.2 It is undisputed that document D2 may be considered as the closest state of the art in the context of the established "problem-solution approach" for the assessment of whether an invention involves an inventive step (Case Law of the Boards of Appeal of the European Patent Office, 9th edition 2019 ("Case Law"), I.D.2) (see impugned decision, point 2.1.1, and statement of grounds of appeal, page 2, lines 11-12).

3.3 Document D2 discloses a method for converting at least one image of a first spectrum into an image of a second spectrum (see abstract, first sentence).

The method comprises a step of recording at least one first-spectrum multi-band reference image of at least one reference scene (see section 2: "fused false colour RGB image"; and section 3: "Let the input multiband nightvision image be the source image") with a first-spectrum night-vision recording apparatus (see section 2: "dual-band visual intensified (DII) camera ... thermal middle wavelength band (3-5 mym) infrared (IR) camera"). The first-spectrum multi-band reference image comprises first image portions with corresponding first-spectrum sensor reference data, wherein the first-spectrum sensor reference data comprises multi-band data, relating to at least two different parts of the spectrum (section 2: "short (visual) wavelength part ... mapped to the R channel", "long (near infrared) wavelength band... mapped to the G channel" , "assigning the IR image to the (empty) B channel").

The method disclosed in document D2 further comprises a step of providing corresponding second-spectrum reference information and of converting the first-spectrum image multi-band reference image into a second-spectrum image (see the first two indents of section 3). More specifically, the "fused false colour RGB image" is transformed into a perceptually decorrelated colour space (see paragraph 3.1). In this decorrelated colour space, the first-order statistics of a natural colour image - whose composition must resemble the composition of the multi-band image (see page 156, left-hand column, lines 4 to 14) - are transferred to the multi-band image. The decorrelation transformation is then reversed to obtain the converted image.

3.4 It is undisputed that document D2 does not disclose the step of providing at least one set of reference data from at least part of first-spectrum multi-band sensor reference data relating to a reference scene and at least a corresponding part of second-spectrum information relating to that reference scene, or the step of converting a target first-spectrum image into a respective second-spectrum image using the set of reference data (see impugned decision, points 2.1.1 and 2.1.2, and statement of grounds of appeal, page 2, last paragraph).

3.5 In the statement of grounds of appeal (page 3, second paragraph), the appellant submitted that the invention aimed to provide a method and system which can provide improved second-spectrum images in a relatively simple, swift manner.

3.6 The board has not been convinced that this effect is achieved by the features of claim 1. The board notes that the claim does not specify how the set of reference data is used to convert an image. Hence, any processing of the reference data, however complex it may be, is covered. It is therefore not apparent how a "simple, swift" conversion is achieved.

3.7 The board therefore formulates the objective technical problem as finding an alternative way of converting the "fused false colour RGB image" disclosed in document D2 into an image with natural daytime colour appearance.

3.8 In point 2.1.4 of the impugned decision, the examining division argued that the skilled person was well aware that lookup tables were used for colour conversion in various applications. The board understands that, in the examining division's view, the use of a lookup table for converting colours was part of the common general knowledge of a person skilled in the art.

3.9 The board shares the examining division's view. The board refers in particular to document D3, which states (page 1, third paragraph) that "[i]t is common to utilize a lookup table (or palette) to indirectly define the colour of the pixels for display (indirect mode)" and that the use of a lookup table provides a number of advantages, for example "colours on the display may be adjusted by adjusting the data content of the palette". The indirect mapping disclosed in the application forming the basis of the appeal in this case is therefore part of the common general knowledge of a person skilled in the art.

3.10 A skilled person starting from document D2 and faced with the above-formulated objective technical problem would therefore consider implementing such an indirect lookup table in order to map "fused false" colours to natural daytime colours. In doing so, the skilled person would necessarily be faced with the problem of how to fill the palette of natural daytime colours so that a meaningful mapping to the "fused false" colours is achieved.

3.11 A meaningful mapping can only be constructed if a priori knowledge is collected about how a "fused false" colour normally maps to a natural daytime colour. Capturing pairs of reference images of relating scenes to collect this knowledge is, in the board's view, obvious.

3.12 In its statement of grounds of appeal, the appellant did not contest that the use of a lookup table for converting colours was part of the general knowledge of the skilled person. With reference to the third auxiliary request (page 7, second full paragraph), it however argued that document D3 led away from the present invention because it concerned the problem of accommodating the difference in the colours supported by the various types of display devices.

3.13 However, the board only refers to document D3 to support its statement that the use of a lookup table for colour conversion is part of the common general knowledge of a person skilled in the art. The board's reasoning is not based on a combination of the inventions disclosed in documents D2 and D3. Thus, this argument by the appellant does not address the point made by the board.

3.14 The board has therefore arrived at the conclusion that the impugned decision was correct in concluding that the subject-matter of claim 1 of the main request lacked inventive step in view of the disclosure of document D2 combined with the common general knowledge of a person skilled in the art.

4. First and second auxiliary requests, inventive step (Article 56 EPC 1973)

4.1 Claim 1 of the first auxiliary request differs from claim 1 of the main request in that it additionally specifies that the first spectrum includes infra-red wavelengths and that the second spectrum is a broad visible spectrum.

4.2 Claim 1 of the second auxiliary request differs from claim 1 of the main request in that it additionally specifies that the second-spectrum reference information is "true-colour colour information".

4.3 In document D2, the first spectrum includes infra-red wavelengths (page 156, right-hand column, lines 16-19) and the second spectrum also includes a "broad" visible spectrum ("normal daylight colour", page 156, right-hand column, lines 38-40, and Figure 4). Thus, document D2 also discloses the additional features of claim 1 of the first auxiliary request.

4.4 Moreover, the "normal daylight colour" RGB images shown in Figure 4 of document D2 can also be considered as representing "true-colour" colour information. Thus, document D2 also discloses the additional features of claim 1 of the second auxiliary request.

4.5 From the above, it follows that the distinguishing features, the objective technical problem and its solution in respect of both claim 1 of the first and second auxiliary requests and of claim 1 of the main request are identical. As a consequence, the reasoning laid out in the previous section 3 also applies to claim 1 of the first and second auxiliary requests.

4.6 In the statement of grounds of appeal, the appellant did not provide additional arguments as to why the subject-matter of the claims of these requests involved an inventive step within the meaning of Article 56 EPC 1973.

4.7 The board has therefore arrived at the conclusion that the subject-matter of claim 1 of the first auxiliary request and the subject-matter of claim 1 of the second auxiliary request lack inventive step in view of the disclosure of document D2 combined with the common general knowledge of a person skilled in the art.

5. Third auxiliary request, inventive step (Article 56 EPC 1973)

5.1 Claim 1 of the third auxiliary request differs from claim 1 of the main request in that it comprises the features added to claim 1 of the first and second auxiliary requests and further stipulates that:

(a) the second-spectrum reference information is already known, for example via visual inspection of a reference scene, or is being provided by providing at least one second-spectrum reference image of each reference scene, the at least one second-spectrum reference image comprising second, second-spectrum, image portions with corresponding second-spectrum information;

(b) the set of reference data is being generated by:

- selecting a plurality of first, first-spectrum, image portions and corresponding second, second-spectrum, image portions, such that reference scene image coordinates of a first image portion substantially correspond to reference scene image coordinates of the corresponding second image portion;- indexing the first-spectrum sensor reference data in a multi-band lookup table and indexing the second-spectrum information in a natural colour lookup table, such that each entry in a resulting index refers to the first-spectrum sensor reference data and the corresponding second-spectrum information associated with a selected first image portion and a corresponding second image portion;

(c) in case of conversion of target first-spectrum images into respective second-spectrum images:

- finding a closest match between first-spectrum sensor data associated with each target first-spectrum image portion of a target first-spectrum image and the first-spectrum sensor reference data of the first image portions;

- replacing each first-spectrum target image portion with the second, second-spectrum, image portion, which second image portion is associated with the first image portion that most closely matched that target image portion, the matching being based on the first-spectrum sensor data, to generate the second-spectrum image;

or, vice-versa, in case of conversion of target second-spectrum images into respective first-spectrum images:

- finding a closest match between second spectrum information associated with each target second-spectrum image portion of a target second-spectrum image and the second-spectrum information of the second image portions;

- replacing each second-spectrum target image portion with the first, first-spectrum, image portion, which first image portion is associated with the first image portion that most closely matched that target image portion, the matching being based on the second-spectrum information, to generate the first-spectrum image.

5.2 The obviousness of the first two groups of features ((a) and (b)) identified in the previous point, having regard to D2 combined with the common general knowledge of a person skilled in the art, follows from the board's reasoning in point 3.9. Indeed, to implement a conversion between "fused false" and daytime colours using an indirect mapping, a lookup table of "fused false" colours and a lookup table of daytime colours must necessarily be generated.

5.3 The last group of features ((c)) relates to the problem of how to convert a target first-spectrum value that is not represented in the lookup table.

This problem will arise, for example, if a palette with a reduced number of colours is used or if the value is not represented in the reference "fused false" images captured to create the palette.

The board is of the opinion that approximating the non-represented values, for example using a nearest neighbour (or "closest match"), is an obvious way of solving this problem.

5.4 In the statement of grounds of appeal, the appellant did not provide any arguments in addition to those discussed in point 3.12 above as to why the subject-matter of the claims of the third auxiliary request involved an inventive step within the meaning of Article 56 EPC 1973.

5.5 In view of the above, the board has arrived at the conclusion that the impugned decision was correct in concluding that the subject-matter of claim 1 of the third auxiliary request (second auxiliary request forming the basis of the impugned decision) lacked inventive step in view of the disclosure of document D2 combined with the common general knowledge of a person skilled in the art.

6. Since none of the appellant's requests is allowable, the appeal is to be dismissed.

Order

For these reasons it is decided that:

The appeal is dismissed.