T 0856/16 (Data searching/VÄÄNÄNEN) of 2.12.2019

Summary of Facts and Submissions

I. The applicant (appellant) appealed against the decision of the Examining Division refusing European patent application No. 12197388.7, published as EP 2 629 211. The application was filed as a divisional application of European patent application No. 09168388.8 (published as EP 2 341 450) with filing date 21 August 2009.

II. The contested decision cited the following document:

D1: US 2008/0306924 A1, published on 11 December 2008

III. The Examining Division decided that the subject-matter of claim 1 of the then main request and the then first and second auxiliary requests infringed Article 123(2) EPC, that the then third and fourth auxiliary requests were inadmissible under Rules 137(3) and 116(2) EPC, and that claim 1 of the then fifth and sixth auxiliary requests lacked inventive step in view of document D1. The Examining Division considered some of the claimed features to be non-technical aspects.

Under the heading "Obiter Dicta", the Examining Division remarked that the application as originally filed did not appear to disclose that the expansion of a query with synonyms was an optional feature. Consequently, the basis for certain features appeared to be highly questionable.

IV. In its statement of grounds of appeal, the appellant requested that the decision be set aside and that a patent be granted on the basis of one of the main and the first and second auxiliary requests considered in the contested decision.

V. In a communication under Article 15(1) RPBA accompanying the summons to oral proceedings, the Board expressed its provisional opinion that the claims of all requests then on file infringed the requirements of Articles 84 and 123(2) EPC.

The Board also drew the appellant's attention to the following documents D2 to D5 (which had been introduced by the Examining Division in the appellant's parallel European patent application No. 10742816) and informed the appellant that he should be prepared to discuss inventive step on the basis of any of the documents D1 to D5.

D2: T. Funkhouser, et al., "A Search Engine for 3D Models", ACM Transactions on Graphics, vol. 22, no. 1, pages 83-105, January 2003

D3: US 2008/0080745 A1, published on 3 April 2008

D4: US 2008/0226119 A1, published on 18 September 2008

D5: US 2009/0208116 A1, published on 20 August 2009

VI. By letter of 21 August 2019, the appellant submitted arguments, a main request, first and second auxiliary requests and an "Optional Main request".

VII. In the course of oral proceedings, held as re-scheduled on 11 October 2019, the appellant was heard on relevant issues. He filed a new main request and new first to third auxiliary requests replacing all prior requests. At the end of the oral proceedings, the chairman announced that the decision would be given in writing.

VIII. The appellant's final request was that the decision under appeal be set aside and that a patent be granted on the basis of the main request or the first, second or third auxiliary request, all filed during the oral proceedings.

IX. By letter filed on 16 October 2019, the appellant submitted a brief note on deliberations. The appellant submitted that he wished to record the raising of a procedural defect in the minutes. He argued that, should the Board's preliminary opinion be upheld, this would be a "direct departure" from the "absolute disclosure standard in the EPC".

X. The minutes of the oral proceedings were dispatched to the appellant with a date of 18 October 2019.

XI. Claim 1 of the main request reads as follows:

"A system comprising a mobile subscriber terminal and a search engine stored in a memory medium of at least one server computer, wherein

-a mixed search input query, comprising at least one text term and at least one image, is arranged to be extracted from visual data arranged to be provided by the camera of a mobile subscriber terminal, is received by said search engine,

-at least one document is arranged to be searched using the said mixed search input query comprising the at least one image with at least one server computer by said search engine,

-most relevant search results obtained with the said mixed search input query are arranged to be retrieved to the mobile subscriber terminal by said search engine."

XII. Claim 1 of the first auxiliary request reads as follows:

"A system comprising a mobile computer and a search engine stored in a memory medium of at least one server computer, wherein

-a mixed search input query, comprising at least one text term, is arranged to be extracted from visual data comprising a traffic sign arranged to be provided by the camera of a mobile computer, is received by said search engine,

-at least one document is arranged to be searched using the said mixed search input query with at least one server computer by said search engine,

-most relevant search results obtained with the said mixed search input query are arranged to be retrieved to the mobile computer by said search engine, and said search results are read out."

XIII. Claim 1 of the second auxiliary request differs from claim 1 of the first auxiliary request in that the following text has been added at the end of the claim:

", and speech is arranged to be used as input for search."

XIV. Claim 1 of the third auxiliary request differs from claim 1 of the second auxiliary request in that it replaces the expression "mobile computer" by "mobile phone", in that it adds the expression "comprising the at least one image" after the phrase "using the said mixed search input query" and in that the following text has been added at the end of the claim: ", and the mixed search input query is arranged to be divided into different search queries by the search engine that are executed by different computers."

XV. The appellant's arguments, where relevant to the decision, are discussed in detail below.

Reasons for the Decision

1. Admissibility of appeal

The appeal complies with the provisions referred to in Rule 101 EPC and is therefore admissible.

Procedural issues

2. In the oral proceedings, the debate was closed and it was announced that the decision would be given in writing. In this procedural situation, the Board could have re-opened the debate in exceptional circumstances.

With his letter filed on 16 October 2019, i.e. after the oral proceedings took place, the appellant requested the Board to record in the minutes of the oral proceedings that a procedural defect had been raised.

According to the established case law (see Case Law of the Boards of Appeal of the EPO, 9th edition 2019, V.B.3.6.2 a) and the decisions cited therein), an objection under Rule 106 EPC is additional to and distinct from other statements, such as arguing or even protesting against the conduct of the proceedings or against an individual procedural finding. The objection must be expressed by the party in such a form that the board is able to recognise immediately and without doubt that an objection under Rule 106 EPC is intended. It also must be specific, indicating clearly and unambiguously on which procedural defect the petitioner intends to rely.

The Board did not include the raising of a procedural defect in its minutes of the oral proceedings, as, during the oral proceedings, nothing was submitted by the appellant which could have been understood as relating to a procedural defect. In his letter filed on 16 October 2019, the appellant identified a possible departure from the "absolute disclosure standard in the EPC" as procedural defect, but the Board considers this objection to be directed to a substantive issue. As no procedural defect is identifiable, the Board saw no reason to re-open the debate before giving its decision in writing.

The invention

3. The application relates to data searching, in particular to searching documents from the internet (description as originally filed, page 1, lines 5 to 8).

3.1 Internet search engines and machine translation are known in the prior art. It is also known to expand the terms in a user's query with synonyms. However, according to the application, the prior art methods for searching and translating data make an incorrect isolated decision in the synonym expansion that leads to wrong results in the overall search execution (description, page 1, line 19, to page 2, line 14).

3.2 In a search method according to the application (description, page 8, line 15, to page 9, line 5; Figure 1), the user inputs a search query. The input can take place by typing text, by speech and/or with an image. In accordance with the invention these input methods can be used in a mix: for example, users can write "how to" and take an image of a box on a ship and the written words "how to" and the pattern recognised words "ship" and "box" are combined to form the search query. Likewise, speech and text and image and speech, or in fact all three forms of input, can be used in a mix.

A further embodiment of the invention comprises a car camera and a computer that reads out traffic signs to the driver as they are videoed and/or photographed (description, page 19, lines 18 to 26).

Main request

4. Admissibility

Claim 1 of the main request submitted in the oral proceedings differs from claim 1 of the main request considered by the Examining Division and initially maintained on appeal as follows: The claim no longer relates to an "internet search engine stored in a memory medium of at least one server computer", but to a "system comprising a mobile subscriber terminal and a search engine stored in a memory medium of at least one server computer". This amendment had already been submitted in reply to the Board's summons in the "Optional Main request" and could be treated without difficulty. Consequently, the Board admitted the main request submitted in the oral proceedings into the appeal proceedings.

Added subject-matter

5. The system of claim 1 comprises the following feature:

A a mixed search input query, comprising at least one text term and at least one image, is arranged to be extracted from visual data arranged to be provided by the camera of a mobile subscriber terminal, [and] is received by said search engine

5.1 In the oral proceedings, the Board asked the appellant whether feature A was to be understood in that at least one text term and at least one image were extracted from a photo provided by the camera of a mobile subscriber terminal such as a mobile phone. The appellant agreed to this interpretation of the claim.

5.2 According to the established case law (see Case Law of the Boards of Appeal of the EPO, 9th edition 2019, II.E.1.3.1 and the decisions cited there), any amendment to the parts of a European patent application relating to the disclosure (the description, claims and drawings) is subject to the mandatory prohibition on extension laid down in Article 123(2) EPC and can therefore, irrespective of the context of the amendment made, only be made within the limits of what a skilled person would derive directly and unambiguously, using common general knowledge, and seen objectively and relative to the date of filing, from the whole of these documents as filed. The skilled person may not be presented with new technical information by the amendment.

5.3 In the Board's communication accompanying the summons to oral proceedings (see points 9.3 and 9.4), the Board objected in particular to feature A of claim 1 under Article 123(2) EPC. In particular, it doubted that there was a basis for extracting text terms and images from visual data captured by a mobile phone or the like. On page 8, lines 23 to 25, 30 and 31, the application disclosed the example of extracting from an image showing a ship with a box on board the search terms "ship" and "box". It did not disclose extracting from a photo captured by a mobile phone an image of a ship and the text term "box", for example.

5.4 In his letter of 21 August 2019, page 1, penultimate paragraph, the appellant referred to the following passages of the originally filed application as the basis for independent claim 1: Figure 1 and the associated explanation on pages 8 to 10, and Figure 4 and the associated explanation on page 19.

Moreover, in the oral proceedings the appellant argued that the original application explained on page 8, lines 16 to 20, that it was possible to input an image when preferably pattern recognition was used by the computer to extract query terms. The word "preferably" disclosed two options: a preferred option of using pattern recognition to extract query terms from an image; and a non-preferred option of using an image as input without pattern recognition (the appellant referred to this as the "raw" image). Claim 1 related to the non-preferred option and used the "raw" image data (see also the appellant's letter of 21 August 2019, page 4, second paragraph, to page 6, paragraph 4). When interpreting the application, the skilled person was aware that computers did not "talk" to each other alphanumerically. Hence, the skilled person would understand that there was no alphanumeric input to the search engine.

5.5 As stated in the oral proceedings, the Board considers that a skilled person reading the passages cited by the appellant does not directly and unambiguously derive the information that at least one text term and at least one image are extracted from a photo/image. The description as originally filed discloses that when visual data (such as a photo/image) is used as input, preferably pattern recognition is used. The skilled person would understand that this passage also discloses not using pattern recognition. However, there is no explicit or implicit disclosure of extracting at least one text term and at least one image from visual data. Moreover, as explained in the following, the Board does not consider that the extraction of image data from visual data is directly and unambiguously derivable from the basis cited by the appellant.

5.6 In the example provided in the description as originally filed on page 8, lines 20 to 28, the computer uses pattern recognition and/or OCR (optical character recognition) to identify characters, that is text, in the photo. Hence, this example does not disclose that "raw" image data is extracted from a photo, but that OCR can be used alternatively or additionally to the use of pattern recognition for extracting text terms. Hence, the skilled person would understand that the non-preferred option (without pattern recognition) for extracting information from an image/photo is based on OCR.

The Board considers that this understanding of the above-cited passage is also supported by the description, page 8, line 28, to page 9, line 5, which explicitly relates to "mixed input" searches and discloses that non-text input, i.e. image or speech data, is recognised to identify the relevant search terms and to order the search terms for querying. It is not recognisable from this passage that "raw" image data is used as part of a search query. Thus, the Board does not agree with the appellant, who pointed to the description, page 8, line 32, to page 9, line 1, as the most important basis for feature A.

5.7 A further passage of the original description on page 8, lines 28 to 32, discloses mixing two of the three disclosed input methods (modalities), i.e. speech, text and image: for example, the user provides the text input "how to" and an image of a box on a ship. Then the written words "how to" and the pattern recognised words "ship" and "box" are combined to form the search query. In this example, text and image are used as input modalities, but again only text terms are output as search query. Consequently, this passage does not provide a basis for feature A either.

5.8 The passage of the description relating to Figure 1, page 9, line 7, to page 10, last line, which the appellant also cited, concerns synonym expansion and contextual ranking and is not suitable as a basis for feature A. Furthermore, Figure 1 does not disclose the extraction of an image from visual data.

5.9 The appellant also cited Figure 4 and page 19 of the description as a basis for claim 1. In the oral proceedings, he argued that different countries could have different traffic signs which were read out to the driver. A picture of a traffic sign was compared to other traffic signs of a similar form, for example. The search engine would recognise the photographed traffic sign among the plurality of traffic signs, for example the traffic sign "give way", and would read this out.

5.10 The relevant passage on page 19, lines 18 to 26, which was cited by the appellant in his letter dated 21 August 2019 (page 7, fifth paragraph), reads as follows:

"In some embodiments of the invention it is possible not to use the translation feature between natural languages e.g. (translating English traffic signs into English text then speech). Useful embodiment being for example a car camera and a computer that reads out traffic signs to the driver as they are videoed and/or photographed. It is possible and in accordance with the invention for the user to operate only the text extraction feature, i.e. translating patterns, characters, image, voice, into textual words. In some embodiments of the invention the extracted textual words that correspond to the data are searched by the synonym expansion - contextual ranking inventive technique."

This passage discloses among other things an embodiment comprising a car camera providing images of traffic signs as input. These traffic signs are then read out to the driver by a computer. The skilled person does not directly and unambiguously derive from the cited passage on page 19 that at least one image and at least one text term are extracted from visual data comprising a traffic sign. In particular, the passage does not disclose that a search query comprising a ("raw") image is extracted from visual data. Rather, as in the passages discussed above that relate to Figure 1, only the extraction of text terms from visual data is directly and unambiguously derivable. Hence, the Board is not persuaded by the appellant's arguments.

5.11 In the oral proceedings, the appellant argued that the skilled person would combine the embodiments relating to Figures 1 and 4, as this was explicitly disclosed in the description, page 20, lines 18 to 20. Thus, the skilled person would derive feature A directly and unambiguously from the combination of the passages relating to Figures 1 and 4.

However, the Board is not convinced by this argument. As none of the cited embodiments discloses extracting a mixed search input query comprising at least one image from visual data, the Board does not see that feature A is directly and unambiguously derivable from the combination of these embodiments.

5.12 In view of the above, the subject-matter of claim 1 extends beyond the content of the application as filed (Article 123(2) EPC).

Auxiliary requests - admissibility

6. The first to third auxiliary requests were filed after oral proceedings had been arranged and add features to independent claim 1 which had not been present in any request filed in time. In particular, the appellant added the feature that the visual data comprises a traffic sign.

The problem defined in the appellant's letter dated 21 August 2019 (page 10, last paragraph) was how to deliver photographic multimodal queries and search results in real time to the mobile consumer. In the oral proceedings, the appellant argued that the objective technical problem to be solved by the auxiliary requests was how to reduce collision-related deaths and injuries while increasing navigation accuracy and efficiency in traffic. Consequently, the auxiliary requests might be regarded as a major change of the claimed subject-matter. However, as the Board has no difficulty in deciding on the substance of the auxiliary requests, it exercises its discretion under Article 13(1) and 13(3) RPBA and admits the auxiliary requests into the appeal proceedings.

First and second auxiliary request

7. Added subject-matter (Article 123(2) EPC)

7.1 Claim 1 of the first and second auxiliary request comprises the following feature:

B a mixed search input query, comprising at least one text term, is arranged to be extracted from visual data comprising a traffic sign arranged to be provided by the camera of a mobile computer, [and] is received by said search engine

7.2 In the oral proceedings, the Board informed the appellant that it doubted that feature B was directly and unambiguously derivable from the application as filed.

In the oral proceedings, the appellant referred to the same passages as the basis for feature B as for feature A of the main request. In particular, he argued that the skilled person could derive feature B from the combination of the passage on page 19, lines 18 to 26, of the description relating to reading out traffic signs to the driver, on the one hand, and the embodiments disclosed in combination with Figure 1 on pages 8 to 10 of the description, on the other.

7.3 Claim 1 refers to a mixed search input query, comprising at least one text term, that is extracted from visual data. The term "mixed search input query" can, in the light of the description, only be understood as relating to the mixing of at least two of the three input modalities (text, speech, image; description, page 8, line 28, to page 9, line 5).

7.4 The Board is not convinced that the cited passages in the application as filed provide a basis for the extraction of a mixed search input query from visual data comprising a traffic sign, for the following reasons.

7.5 On page 19, lines 18 to 26, the description of the application as originally filed discloses that traffic signs are videoed and/or photographed. The skilled person does not derive directly and unambiguously from the cited passage on page 19 that a mixed search input query is extracted from visual data. In particular, this passage does not mention a mixed search input query. With respect to the extraction of data from an image, the passage only discloses the extraction of textual words (page 19, lines 21 to 24). There is no explicit or implicit disclosure of extracting data other than text from visual data. Hence, the Board is not persuaded that the cited passage discloses that a "mixed search input query" is extracted from visual data.

7.6 As discussed above for the main request, the Board understands the passage in the description, page 8, line 15, to page 9, line 5, to mean that text terms are extracted from visual data. Consequently, the cited passage on page 19 and the cited passages relating to Figure 1 disclose extracting text terms from visual data. Thus, even when the skilled person combines an embodiment relating to Figure 1 and the cited passage on page 19, lines 18 to 26, it is not directly and unambiguously derivable that a mixed search input query is extracted from visual data.

7.7 In the oral proceedings, the appellant also argued that a further input for the search query could be obtained from user input (typed text or speech, for example).

However, the Board considers that this argument is not supported by the wording of feature B of claim 1, according to which a mixed search input query, comprising at least one text term, is arranged to be extracted from visual data. Consequently, the Board is not convinced by the appellant's argument.

Merely for the sake of completeness, the Board observes that it doubts that a skilled person who is trying to combine the cited passage on page 19 with the further cited passages on pages 8 to 10 would directly and unambiguously derive the information that, in addition to the visual data comprising a traffic sign, text or speech are obtained by means of user input, thus generating a mixed search input query. The skilled person understands the embodiment disclosed in the cited passage on page 19, lines 18 to 26, as disclosing that the visual data comprising a traffic sign is obtained for example by a car camera while the user is driving the car. It is not directly and unambiguously derivable from the cited passages that, in the situation of driving a car, the user provides input such as text or speech to further define a search query for reading out a traffic sign taking the form of visual data. Such a combination of features is not directly and unambiguously derivable from the cited passages relating to Figures 1 and 4 and the vague hints at combining embodiments on page 20, lines 18 to 20.

7.8 Consequently, feature B is not directly and unambiguously derivable from the passages cited by the appellant and the Board concludes that the subject-matter of claim 1 of the first and second auxiliary request extends beyond the content of the application as filed (Article 123(2) EPC).

Third auxiliary request

8. Added subject-matter (Article 123(2) EPC)

8.1 Claim 1 of the third auxiliary request differs from claim 1 of the second auxiliary request with respect to feature B in that it replaces the expression "mobile computer" by "mobile phone".

8.2 When compared with the first and second auxiliary requests, this amendment changes the device that is associated with the camera for taking images from a mobile computer to a mobile phone.

8.3 In the oral proceedings, the Board informed the appellant that it doubted that claim 1 of the third auxiliary request was directly and unambiguously derivable from the application as filed.

As a basis for claim 1 of the third auxiliary request, in the oral proceedings the appellant referred to the same passages in the application as filed as for the first and second auxiliary requests.

8.4 Like claim 1 of the first and second auxiliary requests, claim 1 of the third auxiliary request still specifies that the claimed system extracts a mixed search input query from visual data comprising a traffic sign. As detailed above for the first and second auxiliary requests, there is no basis for extracting a mixed search input query from visual data comprising a traffic sign in the content of the application as filed.

8.5 It follows that the subject-matter of claim 1 of the third auxiliary request extends beyond the content of the application as filed (Article 123(2) EPC).

Conclusion

9. As none of the appellant's requests can form the basis for the grant of a patent, the appeal is to be dismissed.

Order

For these reasons it is decided that:

The appeal is dismissed.