T 1554/10 (Image rendering using 2D code/SONY) of 17.12.2015

Summary of Facts and Submissions

I. The appeal lies from the decision of the Examining Division to refuse European patent application No. 99305174.7 for lack of inventive step, Articles 52(1) and 56 EPC, of the claimed subject-matter over document D1 in combination with document D2:

D1: US 5 742 263, published on 21 April 1998;

D2: WO 98/24050, published on 4 June 1998.

II. In the statement of grounds of appeal the appellant requested that the decision be set aside and that a patent be granted on the basis of the claims considered in the appealed decision, as a main request, or the claims of a first auxiliary request submitted with the grounds of appeal.

III. The appellant was summoned to oral proceedings. In a subsequent communication sent in advance of the oral proceedings, the Board introduced the following document into the proceedings:

D3: Rolland, J. P. et al.: "A comparison of optical and video see-through head-mounted displays", pages 293 to 307, SPIE Vol. 2351 Telemanipulator and Telepresence Technologies, 1994.

The Board expressed its preliminary opinion that neither of the requests fulfilled the requirements of Article 123(2) EPC, and that the subject-matter of claim 1 of both requests appeared to lack inventive step over the disclosure of document D1.

IV. With a letter of reply the appellant filed a new main request and maintained its previous main and auxiliary requests as first and second auxiliary requests, respectively.

V. During the oral proceedings, which were held on 17 December 2015, the appellant submitted a new main request to replace the previous one and withdrew the auxiliary requests. At the end of the oral proceedings, the chairman pronounced the Board's decision.

VI. The appellant's final request was that the contested decision be set aside and that a patent be granted on the basis of the request consisting of claims 1 to 10 filed as replacement main request in the oral proceedings before the Board.

VII. Claim 1 of the sole request reads as follows:

"An image processing apparatus comprising:

image acquiring means (23) for acquiring a target image (100) including an identification information corresponding to a two-dimensional code (101);

identification information recognizing means (56D) for recognizing said identification information from said target image acquired by said image acquiring means;

target image displaying means (201) for displaying said target image acquired by said image acquiring means (23) inside a specific display area;

position information detecting means (56E) for detecting a plurality of position information, each position information being a respective one of a plurality of two dimensional coordinates making up four corners of said two dimensional code within said target image acquired by said image acquiring means, said two dimensional code being rectangular;

activating means (56F) for activating selectively from among a plurality of previously stored processes a specific process corresponding to said identification information recognized by said identification information recognizing means so as to start execution of said specific process; and

rendering means (52) for rendering an image corresponding to said identification information recognized by said identification information recognizing means (56D), inside said specific display area and in a position calculated in correspondence with said position information detected by said position information detecting means (56E), wherein the coordinates denoting four corners of the image corresponding to said identification information are obtained in correspondence with the coordinates making up the four corners of the two dimensional code."

VIII. Independent claim 8 reads as follows:

"An image processing method comprising the steps of:

acquiring a target image including an identification information corresponding to a two-dimensional code;

recognizing said identification information the acquired target image [sic]; and

displaying said target image acquired by said image acquiring means inside a specific display area;

detecting a plurality of position information, each position information being a respective one of a plurality of two dimensional coordinates making up four corners of said two dimensional codes within said acquired target image, said two dimensional code being rectangular;

activating selectively from among a plurality of previously stored processes a specific process corresponding to said identification information recognized in said identification information recognizing step so as to start execution of said specific process; and

rendering an image corresponding to said identification information recognized in said identification information recognizing step, inside said specific display area and in a position calculated in correspondence with said position information detected in said position information detecting step, wherein the coordinates denoting four corners of the image corresponding to said identification information are obtained in correspondence with the coordinates making up the four corners of the two dimensional code."

IX. Independent claims 9 and 10 read respectively as follows:

"A computer program containing computer readable instructions which, when loaded onto a computer, configures the computer to perform a method according to claim 8."

"A storage medium configured to store the computer program of claim 9 therein or thereon."

Reasons for the Decision

1. The appeal complies with the provisions referred to in Rule 101 EPC and is therefore admissible.

The invention

2. The application is directed to an apparatus for selectively activating a process corresponding to a code in an acquired image, and to render an image corresponding to the code.

The apparatus acquires an image of a target object, for example a card with a two-dimensional bar code ("2D code" in the application) printed on it which the user holds in front of the camera. After acquiring the target image, the apparatus recognises identification information corresponding to an image pattern (e.g. the 2D code) from the target image. It then activates the specific process corresponding to the identification information to start its execution (see page 22, last full paragraph, to page 25, first line, and Figure 11 of the application as originally filed).

In the embodiment discussed on pages 28 and 29, the apparatus displays the target image in a specific display area and renders an image corresponding to said identification information inside that specific display area in a position calculated in correspondence to the position of the image pattern in the target image (see also Figures 12 and 14). The rendered image corresponding to the identification information may be, for example, an icon, an animation, or a facial image (page 28, last full paragraph, page 31, first full paragraph).

Clarity and added subject-matter

3. The Board is satisfied that the text of claim 1 clearly defines the matter for which protection is sought.

4. The combination of features of claim 1 can be directly and unambiguously derived from original claims 1, 3 and 4, and from the passages of the description on page 23, last full paragraph, and on page 28, last full paragraph to page 29, first paragraph.

The last paragraph of page 23 discloses the position information consisting of two dimensional coordinates making up four corners of the two dimensional code, as defined in the claim. The passage on page 28, last full paragraph, discloses, with reference to Figures 12, 14 and 15, that in the method of the invention, the rendered image, such as a face image, is "superimposed onto, say, a blue rectangular background portion ... on the left hand side of the object 100 shown in Fig. 12". The object referred to is the card, or target object, whose image is captured by the system and shown on the display. The next paragraph bridging to page 29 explains that "the coordinates denoting four corners of the background portion ... of the object 100 ... are obtained in correspondence with the 2D code coordinate data". These passages therefore support the features recited at the end of the claim.

The Board hence finds that claim 1 does not add subject-matter within the meaning of Article 123(2) EPC.

Inventive step - Article 56 EPC

5. Document D1 discloses a head-mounted display system to be used in augmented reality or virtual reality applications (column 1, line 50 to column 2, line 3).

5.1 The apparatus of document D1 is a head-mounted display system, whereas the main embodiment of the present invention is described in the application as relying on a conventional personal computer equipped with a video camera mounted on the display (page 11, last full paragraph to page 16, Figures 3 to 8). However, the claims are generally directed to an "image processing apparatus" and do not define any feature limiting the apparatus to a conventional personal computer. They hence also cover a head-mounted system. At the oral proceedings the appellant agreed with this interpretation of the claim.

Consequently, the Board considers it appropriate to base its inventive-step analysis on a comparison with document D1. The specific nature of the device of document D1 has nevertheless to be taken into account, in particular when judging whether the skilled person would modify the apparatus of D1 in a specific manner.

6. The head-mounted display system is described, in document D1, column 1, line 50 to column 2, line 1, as follows:

"More particularly, the head mounted display system of the present invention includes a frame to be supported on a user's head. A display that depicts video information is mounted on the frame as is an optical system for projecting an enlarged image of the displayed video information at a predetermined distance from the user. A barcode sensor is mounted on the frame in a fixed relationship with respect to the display and optical system so that the image detected by the sensor can be mapped onto the displayed image. The barcode sensor detects one or more barcodes in a field of view of the sensor. A processor decodes a barcode detected in the barcode sensor's field of view and the processor controls the video information depicted on the display in accordance with a decoded barcode.

A relatively simple barcode can uniquely identify a particular position in the real world and is easily distinguishable from objects in the real world so as to greatly simplify the detection thereof and the determination of the location of the barcode in the sensor's field of view."

It follows that document D1 discloses an image processing apparatus comprising image acquiring means and position information detecting means for detecting bar code position information, similar to those defined in claim 1.

6.1 Document D1 further discloses that when bar codes are detected in the field of view of the sensor (for example, bar codes 58 to 60 of Figure 2) the microprocessor of the system decodes the detected bar code and controls the information depicted on the display. For example, it depicts human-readable information that is associated with a detected bar code on the display of the head-mounted display system (column 3, line 64 to column 4, line 65), the information being positioned on the displayed image in accordance with the locations of one or more detected bar codes (column 3, lines 5 to 13, column 4, lines 56 to 61).

The apparatus of document D1 therefore comprises identification information recognising means for recognising a bar code, and rendering means for rendering an image corresponding to the bar code, analogous to those recited in claim 1 (see section VII.VII. above). In this respect, the Board generally agrees with the analysis of the Examining Division.

6.2 While the Board is of the opinion that it can be immediately recognised that the apparatus of document D1 comprises "image displaying means", it is not so straightforward to establish whether those means correspond to "target image displaying means" for displaying the acquired target image in a specific display area, as recited in claim 1.

In particular, the Board notes that the head-mounted display system of D1 is built in such a way that the image of the real world can be directly seen through the optical system (column 3, lines 9 to 32, Figure 2), a technique known as "optical see-through" (see document D3, page 293, Introduction, second paragraph). In that case, there is no need to display an image obtained by the camera (corresponding to displaying a "target image" acquired by the "image acquiring means" in the wording of the claim) inside a specific display area, as foreseen by the invention of the present application.

According to the contested decision, the feature "target image displaying means" was disclosed in column 4, lines 49 to 61. However, that passage discloses displaying human readable information, which is information corresponding to the bar code, not a "target image" acquired by the image acquiring means, as recited in the claim. Taking into account the above findings and, additionally, that the Board could not find any other passage of document D1 disclosing the system displaying an acquired target image, the Board considers the feature "target image displaying means", as specified in the claim, as distinguishing the apparatus of claim 1 from that of document D1.

In the Board's view, however, the system of document D1 has means for providing the functionality of the target image displaying means, since it includes a camera for obtaining an image of the real world, processing means for processing the image obtained by the camera and for creating video information, and an optical system for displaying that (or other) video information (column 2, line 54 to column 3, line 32).

Furthermore, at the date of priority of the present application, there were two well-known techniques to implement augmented reality: optical and video see-through (see document D3, page 293, title, abstract, Introduction, first two paragraphs). A video see-through system combines the computer-generated images with the video image of the real world captured by the camera and displays the combined image. The skilled person would hence assume that the system of document D1 could likewise be used for optical and video see-through. The Board thus finds that it would be obvious to adapt the image displaying means to target image displaying means in the apparatus of document D1.

7. The subject-matter of claim 1 further differs from the apparatus of document D1 in that it comprises the following features:

(a) the bar code of the target image is a two-dimensional code;

(b) activating means for activating selectively from among a plurality of previously stored processes a specific process corresponding to the code's identification information;

(c) the plurality of position information (used to calculate the position of the image corresponding to the code's identification information) relates to a single code (instead of a plurality of codes);

(d) each position information is a respective one of a plurality of two-dimensional coordinates making up four corners of the two-dimensional code, said two-dimensional code being rectangular;

(e) the coordinates denoting four corners of the image corresponding to said identification information are obtained in correspondence with the coordinates making up the four corners of the two-dimensional code.

Features (a) to (c) were identified in the contested decision. Features (d) and (e) were added by the last amendments leading to the present claim.

8. At the oral proceedings the appellant argued that by using the four corner points of a single bar code it was possible to calculate more accurately the position for the image to be rendered inside the display area, superimposed on the target image of the real world acquired by the camera. Besides, this was achieved using a single bar code instead of four. Starting from document D1 it would not be obvious to take four points of a bar code because the bar codes in document D1 were one-dimensional.

The Board agrees with the appellant that by detecting the four corners of a rectangular two-dimensional code, and using the four points to calculate the four corners of the rendered image, it is possible to more accurately adapt the presentation of the rendered image to the orientation and position of the two dimensional bar code in the real world. The system may take into account distance, tilt or rotation of the bar code plane in relation to the image plane of the camera. This is useful, for example, when the bar code is printed on a target object such as a card. Assuming that the image corresponding to the two-dimensional code is a photo, the four corners for positioning the photo can be calculated in such a way that the photo appears on the image of the card as if it were printed on the card of the real world.

The Board notes that document D1 discloses in column 5, line 59 to column 6, line 32, a solution to the problem of positioning a symbol in the projected image which, according to the document, takes into account the skew of the angle of the camera or sensor with respect to four bar codes, as shown on Figures 5A and 5B. However, this embodiment of document D1 relies on four bar codes instead of one, and calculates a single position for displaying the symbol on the display, the intersection of the diagonal lines, instead of four corners of an image to be rendered on the display. The rendered image, the symbol, is not tilted. Also, in order to be able to obtain an effect similar to that of the present invention, the four bar codes of document D1 would have to be positioned in the same plane of the real world and form a rectangle. To achieve the above described effect with respect to an image of a target object such as a card, the four bar codes would have to be printed on the same planar surface of the target object. None of these options is mentioned in document D1.

9. The distinguishing features therefore solve the problem of improving the rendered image, taking into account the orientation and position of a target object in relation to the image acquiring means.

In the opinion of the Board this effect is technical, since it relates to adapting the resulting image to reflect real-world spatial conditions captured by the apparatus.

10. The Board shares the appellant's opinion that it would not be obvious for the skilled person to obtain four corners of a single bar code of the system of D1, which only refers to linear or one-dimensional bar codes (see Figures 2, 4A, 4B, 5A and 5B, column 7, lines 41 to 67, Figure 7). Furthermore, none of the embodiments of document D1 addresses the above-mentioned problem. The purpose of the embodiment of column 6 is simply to find a position for displaying a symbol, as explained under point 8 above. In the remaining embodiments of document D1, bar codes are used for head tracking, as indicated from the passage starting in column 6, line 63.

None of the other cited prior-art documents discloses distinguishing features (a) and (c) to (e) in the context of solving problems related to improving rendering, including positioning, of an image within an acquired image of the real world.

The Board therefore concludes that the subject-matter of claim 1 involves an inventive step as required by Articles 52(1) and 56 EPC.

Remittal

11. Since the subject-matter of independent claim 1 is inventive and complies with Article 123(2) EPC, the Board decides to set aside the decision under appeal and remit the case to the department of first instance.

As mentioned at the oral proceedings, the Board is however of the opinion that the reference signs in claim 1 still need to be reviewed when preparing the claims for granting.

12. Regarding the other independent claims, the Board notes the following.

Independent claim 8 recites a method comprising steps corresponding to the apparatus features of claim 1. With the exception of the sentence "recognizing said identification information the acquired target image", and assuming that it should read "recognizing said identification information from the acquired target image", the Board finds that claim 8 clearly defines its subject-matter.

Independent claims 9 and 10 are directed to a computer program and a storage medium defined by reference to claims 8 and 9 respectively.

For the same reasons as given for claim 1, the Board is of the opinion that claims 8 to 10 fulfil the requirements of Articles 123(2) and 56 EPC.

The Board therefore concludes that prior to grant, the above-mentioned sentence of independent claim 8 should be corrected.

13. The dependent claims and description may also have to be adapted.

Order

For these reasons it is decided that:

1. The decision under appeal is set aside.

2. The case is remitted to the department of first instance for further prosecution.