T 1080/12 () of 4.9.2018

Summary of Facts and Submissions

I. The appeal is directed against the decision to refuse European patent application No. 08 154 350.6, published as EP 1 986 441 A2.

II. The examining division refused the application on the grounds that the independent claims of the main request were not clear and that the claimed subject-matter of the main request and the auxiliary request did not involve an inventive step in view of the following documents:

D11: Sugiyama K; Aoki T; Hangai S: "A picture rate conversion scheme using fully motion compensated interpolation", Conference Proceedings Article, 2005 Digest of Technical Papers, International Conference on Consumer Electronics (IEEE CAT. NO.05CH37619) IEEE Piscataway, NJ, USA, 1 August 2005, pages 75 and 76, XP010796538 and

D12: Kuo Tien-Ying; Kim JongWon; Kuo C-C Jay: "Motion-compensated frame interpolation scheme for H.263 codec", Conference Proceedings Article, Proceedings of the 1999 IEEE International Symposium on Circuits and Systems/ISCAS '99, 30 May to 2 June 1999, Orlando, Florida, 30 May 1999, IEEE Service Center, Piscataway, NJ, Vol. 4, pages 491 to 494, XP010341210.

III. The applicant/appellant filed notice of appeal against this decision and with its statement of grounds of appeal submitted claims of a new auxiliary request.

IV. On 24 November 2017, the board issued a communication pursuant to Rule 100(2) EPC raising several objections as to lack of clarity and inventive step against the claims of the main and auxiliary requests then on file.

V. In letters dated 26 March 2018 the appellant replied to the board's communication and submitted amended claims according to a main request and first and second auxiliary requests replacing the claims then on file.

VI. The board summoned to oral proceedings and indicated in a communication annexed to the summons inter alia that it considered the subject-matter of the independent claims of the main and first auxiliary requests then on file to lack inventive step in view of document D11 in combination with document D12. It also stated that it was inclined to remit the case to the department of first instance with the order to grant a patent on the basis of the claims of the second auxiliary request and a description to be adapted thereto.

VII. In a letter dated 13 June 2018, the appellant withdrew the then pending main and first auxiliary requests. It requested that a European patent be granted on the basis of the application as amended according to the pending second auxiliary request, which became its sole request.

VIII. The board subsequently cancelled the oral proceedings.

IX. The claims of the present sole request read as follows:

"1. An image processing apparatus for performing frame rate conversion on an input image signal comprising a chronological sequence of image frames, the apparatus comprising:

a detecting unit (92) configured to detect motion between successive image frames of the input image signal, on a block-by-block basis, and to output a motion vector indicative of the detected motion;

a determining unit (82) configured to determine whether said input image signal is subject to fade in or fade out; and an interpolating unit (95) configured:

responsive to a determination by the determining unit that the input image signal is not subject to fade in nor to fade out, to generate and output an intermediate image frame interposed chronologcally [sic] between a first image frame of said input image signal and the preceding image frame of said input image signal, by interpolation between said first image frame and said preceding image frame in accordance with said block by block motion vector, and

responsive to a determination by the determining unit that the input image signal is subject to fade in or to fade out, to allow said preceding image frame of the input image signal to be output unchanged as said intermediate image frame interposed chronologcally [sic] between said first image frame and said preceding image frame;

wherein the determining unit (82) is configured to perform processing to detect fade-in or fade-out only if the number of pixels for which either said first image frame of the input image signal or said preceding image frame of the input image signal is at a zero level is larger than a threshold value, then said determining unit (82) determines whether said input image sgnal [sic] is subject to fade-in or to fade-out based on a time-varied change in at least one of factors representing a maximum level of said input image signal, the number of pixels for which said input image signal is at a zero level, and a total sum of all pixel levels in respective successive frames of said input image signal.

2. An image processing method for use with an image processing apparatus for performing frame rate conversion on an input image signal comprising a chronological sequence of image frames, said image processing method comprising the steps of:

detecting motion between successive image frames of said input image signal, on a block-by-block basis, and outputting a motion vector indicative of the detected motion;

determining whether said input image signal is subject to fade in or fade out; and

upon determining, in the determining step, that the input image signal is not subject to fade in nor to fade out, generating and outputting an intermediate image frame interposed chronologcally [sic] between a first image frame of said input image signal and the preceding image frame of the input image signal, by interpolation between said first image frame and said preceding image frame in accordance with said block by block motion vector, and

upon determining, in the determining step, that the input image signal is subject to fade in or to fade out, allowing said preceding image frame of the input image signal to be output unchanged as said intermediate image frame interposed chronologcally [sic] between said first image frame and said preceding image frame;

wherein the determining performs processing to detect fade-in or fade-out only if the number of pixels for which either said first image frame of the input image signal or said preceding image frame of the input image signal is at a zero level is larger than a threshold value, then said determining determines whether said input image sgnal [sic] is subject to fade-in or to fade-out based on a time-varied change in at least one of factors representing a maximum level of said input image signal, the number of pixels for which said input image signal is at a zero level, and a total sum of all pixel levels in respective successive frames of said input image signal.

3. A program configured for causing a computer to perform frame rate conversion on an input image signal comprising a chronological sequence of image frames, said image processing procedure comprising the steps of:

detecting motion between successive image frames of said input image signal, on a block-by-block basis, and outputting a motion vector indicative of the detected motion;

determining whether said input image signal is subject to fade in or fade out; and

upon determining, in the determining step, that the input image signal is not subject to fade in nor to fade out, generating and outputting an intermediate image frame interposed chronologcally [sic] between a first image frame of said input image signal and the preceding image frame of the input image signal, by interpolation between said first image frame and said preceding image frame in accordance with said block by block motion vector, and

upon determining, in the determining step, that the input image signal is subject to fade in or to fade out, allowing said preceding image frame of the input image signal to be output unchanged as said intermediate image frame interposed chronologcally [sic] between said first image frame and said preceding image frame;

wherein the determining performs processing to detect fade-in or fade-out only if the number of pixels for which either said first image frame of the input image signal or said preceding image frame of the input image signal is at a zero level is larger than a threshold value, then said determining determines whether said input image sgnal [sic] is subject to fade-in or to fade-out based on a time-varied change in at least one of factors representing a maximum level of said input image signal, the number of pixels for which said input image signal is at a zero level, and a total sum of all pixel levels in respective successive frames of said input image signal."

X. The examining division's objections where relevant for the present claims may be summarised as follows:

It was unclear how a "block by block motion vector" differed from a usual motion vector. The moving section defined in the description, page 26, lines 27 and 28, was an essential feature for the mixing of the interpolation target signal and should be specified in the claim.

D11 was considered the closest prior art with respect to the claimed subject-matter. It disclosed all features of the then claim 4 except for the input image signal being output unchanged as said input image signal intermediate signal if said input image signal was subject to fade-in or fade-out. The corresponding technical problem was what method to perform when fade-in/out was detected. The distinguishing feature was disclosed in D12, which the skilled person would have combined with D11 to arrive at the claimed subject-matter.

Reasons for the Decision

1. The appeal is admissible.

2. The present invention relates to an image processing method for frame rate conversion as well as a corresponding apparatus and program. According to a conventional method for frame rate conversion which is acknowledged in the application, motion vectors between successive input frames are detected in order to interpolate an image signal at certain points in between the input frames. However, when using this prior-art method, the image quality of interpolated/intermediate frames was degraded if the input frames were subject to fade-in or fade-out, i.e. if the image sequence described a transition to or from a blank image (see page 1, line 16, to page 2, line 27, of the application).

The present invention improves on the conventional method by determining whether an input image sequence is subject to fade-in or fade-out. If this is the case, the preceding image of the input image signal is output unchanged as an intermediate image frame (frame repetition). In all other cases the conventional method of interpolation in accordance with the detected motion vectors is applied. The determination of fade-in and fade-out is performed only if the number of zero-level pixels for one of two subsequent image frames is larger than a threshold value. Fade-in or fade-out sequences are then detected using at least one of the following criteria: a maximum level of the input image signal, the number of pixels for which the input signal is at a zero level or the total sum of all pixel levels in successive frames of the input image signal (see page 6, line 21, to page 7, line 5; page 10, lines 6 to 13; page 11, lines 2 to 9, and figures 8 to 10).

Amendments (Article 123(2) EPC)

3. Compared with claim 1 of the main request underlying the decision under appeal, present claim 1 has been amended to specify that the intermediate frame is "interposed chronolog[i]cally between a first frame of said input image signal and a preceding frame of said input image signal". Claim 1 now also specifies that the outputting of the intermediate image frame is "responsive to a determination by the determining unit" and that the "block by block motion vector" is used in the interpolation of an "input image signal intermediate signal". It further specifies that the detecting unit is configured to detect motion between successive image frames of the input image signal on a "block-by-block basis". In addition, it defines the criteria for deciding on whether a fade-in/out is present and specifies that fade-in/out detection is carried out only if the number of pixels for which either the first image frame or the preceding image frame "of the input image signal is at zero level is larger than a threshold value". Corresponding amendments have been made to the further independent claims 2 and 3. In addition, the claims have been reworded.

A basis for these amendments can be found in the application as filed on page 1, lines 25 to 29, together with page 13, lines 8 to 11; page 10, lines 19 to 25, together with page 11, lines 2 to 14; page 12, lines 9 to 24. The criteria for deciding whether a fade-in/out is present and whether fade-in/out detection should be carried out are disclosed in claims 2 and 3 as originally filed. Hence, the board finds that the claims of the appellant's sole request do not contain subject-matter which extends beyond the content of the application as filed and that they thus comply with Article 123(2) EPC.

Clarity (Article 84 EPC)

4. According to the decision under appeal it was unclear how a "block by block motion vector" differed from a usual motion vector. Furthermore, the moving section defined in the description, page 26, lines 27 and 28, was an essential feature for the mixing of the interpolation target signal and should be specified in the claim (see point VIII above).

4.1 Amended claim 1 now specifies that the detecting unit is configured to detect motion between successive image frames of the input image signal on a "block by block basis". In addition, according to claim 1 the "block by block motion vector" is used in the interpolation of the intermediate image frame. Corresponding amendments have been made to the further independent claims 2 and 3.

4.2 The board holds that the person skilled in the art would unambiguously understand from amended claim 1 that the motion vectors are determined on a block-by-block basis for successive frames, thereby matching blocks for the first image frame and the preceding image frame. Such determination is well known as block matching and corresponds to the motion estimation of the conventional art cited in the application (see page 2, lines 11 to 19, and page 12, lines 9 to 14).

4.3 The moving section 93 is described on page 12, lines 15 to 18, and on page 25, lines 27 to 29, as moving "the preceding interpolation target signal coming from the frame memory 91 on a block-by-block basis through the use of the block-by-block motion vector". The moving section therefore supports the interpolation using motion vectors.

The invention is based on conventional interpolation using motion vectors, which is specified in terms of functional features in claim 1 ("generate and output an intermediate image frame ... by interpolation ... in accordance with said block-by-block motion vector"). The invention concerns an alternative method (frame repetition) which is applied if a fade-in/out is detected. The moving section is a component of the conventional part of the image processing apparatus and not a component of the improvement provided by the application and consequently not an essential feature of the invention.

4.4 It follows that the clarity objections raised by the examining division either have been overcome by the amendments to the present claims or are not justified. Nor is there any other respect in which the board considers the present set of claims to lack clarity. Hence, the present claims meet the requirements of Article 84 EPC.

Inventive step (Article 56 EPC)

5. It is common ground that D11 may be considered the closest prior art with respect to the present application.

5.1 D11 discloses a frame interpolation method in which a target frame is interpolated from prediction blocks designated by motion vectors, which are searched from two consecutive frames. It discloses experimental results with good PSNR except during "fading out objects" (see figure 1 and chapters "The Proposed Interpolation" and "Experimental Results").

5.2 D11 does not disclose the use of frame repetition during fade-in/out image sequences. It also fails to show the criteria for fade-in/out detection, specifically determining whether the number of zero-level pixels for one of two consecutive frames is larger than a threshold value and subsequently testing for fade-in or fade-out based on a change either in the maximum level of the input image signals or in the number of zero-level pixels of the input image signal or in a total sum of all pixel levels in successive frames.

5.3 The objective technical problem in view of these distinguishing features can be formulated as how to improve the image quality of interpolated images which may be subject to fade-in or fade-out and to provide appropriate steps for detecting fade-in/out.

5.4 D11 may be construed as providing a hint that improvement was needed for fade-in/out image sequences. D12, which also concerns frame interpolation, proposes using frame repetition if two adjacent encoded frames exhibit a significant difference in intensity. The applicable criterion is that the averaged pixel intensity change along the motion trajectory is larger than a threshold (see chapter 4).

This criterion for detecting intensity changes in D12 differs from the two-step procedure specified in claim 1, according to which fade-in/out detection is carried out only if the number of zero-level pixels for one of two subsequent image frames is larger than a threshold value (first step). In a second step, fade-in or fade-out sequences are detected using at least one of the following criteria: a maximum level of the input image signal, the number of pixels for which the input signal is at a zero level or the total sum of all pixel levels in successive frames of the input image signal.

5.5 Hence, the subject-matter of claim 1 and the further independent claims is not rendered obvious by the combination of D11 and D12. Nor can the board see any other document or combination of documents on file by which the skilled person would have arrived at the claimed subject-matter.

5.6 It follows that the subject-matter of claim 1 and the further independent claims 2 and 3, which have been restricted by features corresponding to those restricting claim 1, involves an inventive step (Article 56 EPC).

Conclusion

6. The board sees no obstacle to granting a patent on the basis of the present claims. However, the description needs to be duly adapted. Hence, the board considers it appropriate to remit the case to the department of first instance with the order to grant a patent with the present claims, with a description to be adapted thereto.

Order

For these reasons it is decided that:

1. The decision under appeal is set aside.

2. The case is remitted to the department of first instance with the order to grant a patent with the following claims and a description to be adapted thereto:

Claims: Nos. 1 to 3 according to the sole request filed as second auxiliary request by letter of 26 March 2018.