T 0677/17 (Augmented reality, detecting position of apparatus / Sony) of 13.2.2020

Summary of Facts and Submissions

I. The appeal lies from the decision of the examining division to refuse European patent application No. 10818085.2 because the main request and first to fourth auxiliary requests did not meet the requirements of Article 56 EPC. The fifth and sixth auxiliary requests were not admitted into the proceedings under Rule 137(3) EPC.

II. The examining division made reference to the following documents:

D1 EP 1 060 772

D2 US 2005/024388

III. In its statement setting out the grounds of appeal the appellant submitted arguments and two further, seventh and eighth, auxiliary requests.

IV. The board arranged for oral proceedings to be held.

V. In the summons, the board set out its provisional view of the case. It considered, inter alia, that none of the requests met the requirements of Article 83 EPC. With regard to the fifth and sixth auxiliary requests the question of admissibility was raised.

VI. By letter dated 13 January 2020, the appellant submitted arguments and filed an amended main request and the following two scientific papers:

Mittal et al.: "M2Tracker: A Multi-View Approach to Segmenting and Tracking People in a Cluttered Scene", International Journal of Computer Vision 51(3), 189-203, 2003;

Lee et al.: "Convolutional Deep Belief Networks for Scalable Unsupervised Learning of Hierarchical Representations", Proceedings of the 26**(th) International Conference on Machine Learning, Montreal, Canada, 2009.

VII. By letter dated 14 January 2020, the appellant filed a corrected main request.

VIII. The oral proceedings were held on 13 February 2020.

IX. The appellant requested that the decision under appeal be set aside and that a patent be granted on the basis of the main request as filed with letter dated 14 January 2020 or, alternatively, on the basis of any of the first to eighth auxiliary requests, the first to sixth auxiliary requests as filed with letter dated 19 August 2016 during examining proceedings, the seventh and eighth auxiliary requests as filed with the statement setting out the grounds of appeal.

X. Claim 1 of the main request reads as follows:

"An information processing apparatus (10) comprising:

an image acquisition unit (152) for acquiring a real space image including an image of another apparatus (20);

a coordinate system generation unit (156) for generating a spatial coordinate system of the real space image acquired by the image acquisition unit (152);

a transmission unit (166) for transmitting spatial information constituting the spatial coordinate system generated by the coordinate system generation unit to the other apparatus (20) for sharing the same spatial coordinate system;

an other terminal detection unit (158) for detecting the image of the other apparatus (20) included in the real space image and for specifying a spatial coordinate of the other apparatus (20) in the spatial coordinate system based on the real space image; and

an acquisition unit (154) for acquiring identification information of virtual information (51, 52), the spatial coordinate of the other apparatus, a spatial coordinate of its own apparatus, and a rotation angle of a vector comprising yaw, pitch and roll connecting the spatial coordinate of the other apparatus (20) to the spatial coordinate of its own apparatus (10),

wherein the transmission unit (166) transmits the identification information of the virtual information (51, 52) to the other apparatus (20) together with a spatial coordinate of a display position of the virtual information (51, 52) and the spatial coordinate of the other apparatus (20), the spatial coordinate of its own apparatus (10), and the rotation angle of the vector comprising yaw, pitch and roll to the other apparatus (20) as the spatial information."

XI. Claim 1 of the first to eighth auxiliary requests comprises the same definitions of the image acquisition unit and of the other terminal detection unit as claim 1 of the main request:

an image acquisition unit (152) for acquiring a real space image including an image of another apparatus (20);

an other terminal detection unit (158) for detecting the image of the other apparatus (20) included in the real space image and for specifying a spatial coordinate of the other apparatus (20) in the spatial coordinate system based on the real space image.

Reasons for the Decision

The present application pertains to apparatuses and a system for presenting an image comprising real space parts and superimposed virtual parts to two users. The positions of the apparatuses used by the user are detected and taken into account.

Main request

1. Sufficiency of disclosure, Article 83 EPC

The board holds that the application does not comply with the requirements of Article 83 EPC.

1.1 The information processing apparatus of claim 1 comprises "an other terminal detection unit (158) for detecting the image of the other apparatus (20) included in the real space image and for specifying a spatial coordinate of the other apparatus (20) in the spatial coordinate system based on the real space image". This unit carries out two functions:

(a) detecting the image of the other apparatus included in the real space image; and

(b) specifying a spatial coordinate of the other apparatus in the spatial coordinate system based on the real space image.

1.2 The real space image referred to is, for example, "a landscape captured by an imaging apparatus" (paragraph 44). In particular, a two-dimensional digital image, as taken by a digital camera, is covered by the claim's wording.

1.3 The description, in paragraph 55, sets out that "[t]he image of the other terminal included in the real space image, for example, can be detected using a well-known image processing means for performing background difference and the like". Similar wording is repeated in paragraph 82. These passages are the only explanation in the description relating to function (a).

1.4 The application does not contain any further explanation of the "background difference" image processing technique. In the field of image processing "background difference" is mostly used for detecting a moving object in a movie sequence. However, the "other terminal detection unit" as claimed does not have a movie sequence at its disposal, but only a real space image. Moreover, the detection of the image of an apparatus included in a real space image presupposes that the detecting unit possesses information about properties of the apparatus to be detected, e.g. size, form, colour, etc., or alternatively about properties of the background included in the real space image. In the board's view, claim 1 does not specify or limit the properties of the other apparatus or of the background.

1.5 As to feature (b), the board notes that clearly the spatial coordinates of the other apparatus can only be specified once the other apparatus has been detected. In this regard, paragraph 55 contains the following explanation:

"the position of the other terminal in the spatial coordinate system, for example, can be specified using a position detection method disclosed in Japanese Unexamined Patent Application Publication No 2006-209334 and the like".

1.6 According to the abstract, this Japanese patent application pertains to the detection of the position of a human person in a two-dimensional image. Paragraph 61 of the description of this publication suggests that another animal or another moving body can be detected too; however no corresponding details are provided. Hence, this published application does not provide an enabling disclosure for feature (a) or (b) over essentially the whole scope claimed.

1.7 The board is not aware of any common general knowledge which would enable the skilled person to put into practice the features referred to in section 1.1, in particular in view of the very broad wording "image of the other apparatus".

1.8 At the oral proceedings the appellant argued that the line of sight to the other apparatus was known and only the distance from it had to be detected in order to specify the spatial coordinates of the other apparatus.

The board does not accept this argument. As set out above, detecting the image of a not further specified other apparatus in a two-dimensional image taken by one camera is not sufficiently disclosed. Clearly, the image of the other apparatus has to be detected first, before a line of sight can be established.

1.9 Referring to Figure 2 of the unexamined Japanese patent application (see section 1.5 above), the appellant explained how the vertical size and the position on the x-axis of an object could be established. The appellant agreed that some prior knowledge about the other apparatus was needed, such as the shape or colour.

However, claim 1 does not specify any properties of the other apparatus. Furthermore, the unexamined Japanese patent application discloses detecting the position of human-shaped objects, and that other objects could indeed be detected too, but only by using filters "according to the 3D shape of the object". However, no such shape is specified in the claims.

1.10 Referring to the abstract of the paper by Mittal et al., the appellant also argued that when occlusion is minimal, an image taken by a single camera is sufficient to recognise the image of the other apparatus. Furthermore, this paper suggested the use of Bayesian Classification for detecting persons in an image.

These arguments are not convincing. The abstract of the paper by Mittal et al. clearly teaches that multiple synchronised surveillance cameras are used. Furthermore, features of the object to be detected have to be modelled in advance. This appears to be impossible for the "other apparatus" as claimed.

Section 5 of this paper specifically deals with "Pixel Classification in a Single View". In sub-section 5.1 it is stated that "the algorithm as described above assumes that we have information about the people visible in the scene and fails when there are people for whom we do not have any information or have inaccurate information". This explanation confirms that prior knowledge of the other apparatus is needed.

1.11 Referring to the paper by Lee et al., the appellant submitted that by using unsupervised learning based on unlabelled data, a neural network could be trained to recognise images of objects.

However, the board considers that the training data, by necessity, has to include at least some images of objects similar to those to be detected later. In other words, a neural network is not able to recognise the image of an unspecified object. Moreover, the detection result would be highly dependent on the unspecified training data.

1.12 The appellant also argued that the image acquisition unit as claimed could capture a sequence of images, which could be used for separating the foreground from the background, applying the technique "background separation" as suggested in the description of the application.

However, the subject-matter claimed also covers situations in which only one real space image is acquired.

1.13 For the above reasons, the application, including in view of the general knowledge of the skilled person, does not disclose features (a) and (b) in a sufficiently clear and complete manner.

First to fourth and seventh and eighth auxiliary requests

2. Sufficiency of disclosure, Article 83 EPC

The board holds that the application according to these requests does not comply with the requirements of Article 83 EPC, for the same reasons as set out above with regard to the main request.

Fifth and sixth auxiliary requests

3. In the course of the oral proceedings the examining division decided to not admit these requests, pursuant to Rule 137(3) EPC. The appellant did not address this aspect of the examining division's decision in the statement of grounds. The board judges that the examining division exercised its discretion in a proper manner.

For these reasons, the board decides not to admit the fifth and sixth auxiliary requests, based on Article 12(4) RPBA 2007.

Order

For these reasons it is decided that:

The appeal is dismissed.