T 1455/16 (Image search to obtain shooting spot/RAKUTEN) of 20.11.2019

Summary of Facts and Submissions

I. The appeal lies from the decision of the Examining Division to refuse European patent application No. 11775122.2 for lack of inventive step in the subject-matter of the claims of a main request and first to fifth auxiliary requests over prior-art document

D1: US 5 579 471, published on 26 November 1996.

Claims 1 and 7 to 9 of the fifth auxiliary request were considered to lack clarity.

II. In the statement of grounds of appeal, the appellant requested that the decision be set aside and that a patent be granted on the basis of one of the main request or first to fifth auxiliary requests considered in the appealed decision and resubmitted with the grounds of appeal.

III. In a communication accompanying the summons to oral proceedings, the Board expressed its preliminary opinion that the subject-matter of claim 1 of each of the requests lacked inventive step and that claim 1 of the fifth auxiliary request was unclear and might add subject-matter.

IV. In a letter of reply the appellant further argued its case.

V. Oral proceedings were held on 20 November 2019 and attended by the appellant. At the end of the oral proceedings, the chairman pronounced the Board's decision.

VI. The appellant requested that the decision under appeal be set aside and that a patent be granted on the basis of the main request or, in the alternative, on the basis of one of the first, second, third, fourth or fifth auxiliary requests, all requests having been filed with the statement of grounds of appeal.

VII. Claim 1 of the main request reads as follows:

"An information providing device (2) that a terminal device (1-k, k=1,2,3...,n) can access through a network, the information providing device comprising:

a receiving means (21) arranged to receive arrangement information indicating a type of a subject and an arrangement of the subject in a frame, from the terminal device through the network; and

a searching means (23) arranged to compare the arrangement information stored in an image data memory means (22) that stores the image data and the arrangement information indicating the type of the subject and the arrangement of the subject included in the image data, with the arrangement information received by the receiving means, and arranged to search for image data based on a result of the comparison,

wherein the image data memory means is arranged to store position information of the image data, indicating the shooting spot of the image data; and

the type of the subject matches a tab selected among tabs (311) provided per genre of object information;

further comprising a transmitting means (21) arranged to transmit the position information of the image data searched by the searching means, to the terminal device."

VIII. Claim 1 of the first auxiliary request differs from claim 1 of the main request in that "and" before the text "the type of the subject matches [...]" was replaced with:

"the receiving means is arranged to receive condition information indicating a condition related to the position information; and

the searching means is arranged to perform the comparison of the image data matching position information which satisfies the condition indicated by the received condition information;".

IX. Claim 1 of the second auxiliary request differs from claim 1 of the first auxiliary request in that the text "the receiving means [...] by the received condition information;" cited in preceding section was replaced with:

"the receiving means is arranged to receive condition information indicating a condition related to the position information and specified object information of a shooting target; and

the searching means is arranged to perform the comparison of the image data matching position information which satisfies the condition indicated by the received condition information and captured within a predetermined range based on position information of the shooting target;".

X. Claim 1 of the third auxiliary request differs from claim 1 of the first auxiliary request in that the following text was inserted before "the type of the subject matches [...]":

"when a position at which object information is arranged by a user and an arrangement position at which corresponding object information is represented in image data are compared, a range in which a degree of coincidence is decided is gradually expanded;"

XI. Claim 1 of the fourth auxiliary request differs from claim 1 of the second auxiliary request in that the following paragraph was inserted after the text "based on position information of the shooting target;":

"the predetermined range is increased stepwise based on the position information of the shooting target;".

XII. Claim 1 of the fifth auxiliary request differs from claim 1 of the second auxiliary request in that the following text was inserted after the text "based on position information of the shooting target":

", the predetermined range being determined based on a rate of a size of the arrangement information arranged in the frame with respect to the frame".

XIII. The appellant's arguments, where relevant to this decision, are addressed in detail below.

Reasons for the Decision

1. The appeal complies with the provisions referred to in Rule 101 EPC and is therefore admissible.

Invention

2. The present invention concerns retrieving images according to user specified criteria and recommending a shooting spot at which such an image can be captured, where the criteria regards the objects presented on the image and their arrangement on the image area.

2.1 According to the invention, an information providing server or device stores image data and associated image information in a database (paragraphs [0030] to [0032] of the application). The image information includes object information about objects or subjects in the image (e.g. people, animals, plants, landscapes, see paragraph [0034]); arrangement information of each object in the image (areas of the image in which the object is represented, see paragraph [0035] and Figures 4A and 4B); and shooting location (e.g. latitude and longitude or address, see paragraph [0033]).

2.2 At a terminal, a user specifies "arrangement information" indicating "object candidates", each corresponding to a subject type, arranged in a particular manner in a "pseudo frame" representing an image area. Figure 5 illustrates a user interface to specify a search by arranging the position, shape and size of (rectangular) objects, each representing a subject type, on a pseudo-frame window (paragraphs [0040] to [0042]). In the example of Figure 5, the user specifies a search for images showing two mountains, a river and a person in a given spatial arrangement on the image.

The arrangement information is sent to the information providing server, which then searches for image data matching the arrangement information and sends the image data found, optionally with corresponding information about a shooting spot at which an image can be captured according to the arrangement information (paragraphs [0006] and [0007]).

Main request

3. Claim 1 of the main request defines an information providing device that a terminal device can access through a network, the information providing device comprising:

(a) a receiving means arranged to receive arrangement information indicating a type of a subject and an arrangement of the subject in a frame, from the terminal device through the network; and

(b) a searching means arranged to compare the arrangement information stored in an image data memory means that stores the image data and the arrangement information indicating the type of the subject and the arrangement of the subject included in the image data, with the arrangement information received by the receiving means, and arranged to search for image data based on a result of the comparison,

(c) wherein the image data memory means is arranged to store position information of the image data, indicating the shooting spot of the image data; and

(d) the type of the subject matches a tab selected among tabs provided per genre of object information;

(e) further comprising a transmitting means arranged to transmit the position information of the image data searched by the searching means, to the terminal device.

4. Claim interpretation

In functional terms, claim 1 essentially specifies a device for searching images matching conditions described by "arrangement information" which indicates a subject type and an arrangement of the subject in a frame. The device searches images containing an object of the given subject type (e.g. river or person) arranged in the given manner on the image, and then returns position information indicating the shooting spots of the matching images.

Interpreting feature (d) in the light of the description and drawings, especially paragraphs [0040] to [0042] and Figure 5, the claimed device supports tabs, one for each object genre, for example People, Animals, Landscape 1 or Landscape 2. Subject types are classified into object genres. For example, the object genre Landscape 1 shown in Figure 5 includes the subject types House, Building, Mountain, Lake, River, Sea, Forest and Waterfall. The user may select one of those tabs to choose from the corresponding window a user interface element corresponding to "a type of a subject" (e.g. Mountain or River under the tab for Landscape 1) and move it to a position within the pseudo-frame.

5. Inventive step - claim 1

5.1 Document D1 discloses an on-line image database from which image data can be retrieved on the basis of visual characteristics such as colours, textures, shapes, and sizes, as well as textual tags appended to the images (abstract, column 2, lines 36 to 41). Queries are constructed in an image query construction area in an example image window where a user builds a sample image representing the important aspects of the images being sought (abstract, column 8, line 66 to column 9, line 11; Figure 5). The sample image specifies values for image characteristics such as colour, texture or category, where a category can be an arbitrary text tag appended to the image (e.g. "GRASS", "BEARS"). A sample image can be built by dragging characteristic thumbnails, e.g. "BEARS" or "WATER" thumbnails, to the example image window. The thumbnails are placed according to a specific layout which affects the results of the query. For example, the user places two bear thumbnails next to each other in the example image window to indicate a preference for adjoining bear objects in the result set of images to be retrieved (column 9, line 12 to column 10, line 24). A spatial partitioning, such as a rectangular grid, is defined for each image. Using the areas defined by the partition, a set of image characteristics is stored for each area (column 12, line 49 to column 13, line 4).

5.2 In its statement of grounds of appeal, the appellant agreed with the contested decision's conclusion that the features distinguishing the subject-matter of claim 1 from the information providing device of document D1 were feature A, corresponding to features (c) and (e) above, and feature B corresponding to feature (d) above.

The Board, however, is of the opinion that, since the device of document D1 already includes image data memory means and transmitting means, the difference with regard to features (c) and (e) is merely that such means are further arranged to store and transmit position information.

With regard to feature (d), the Board notes that in document D1 the user moves thumbnails to the sample image. The thumbnails can be selected from selection windows (containers) provided for different types of characteristics, e.g. colour, texture, shape or category (column 9, lines 12 to 32). Each element of a category (e.g. "Bears" and "Birds" in Figure 5) corresponds to a subject type of claim 1. However, the Board agrees that document D1 does not disclose tabs and genres of objects (which would correspond to category genres in that document) and, thus, that feature (d), if interpreted in the light of the description, is new over D1.

The subject-matter of claim 1 therefore differs from the information providing device described in document D1 in that it includes the following features:

(A') the image data memory means is further arranged to store position information of the image data indicating the shooting spot of the image data, and the transmitting means is further arranged to transmit the position information of images searched by the searching means (from features (c) and (e)); and

(B) the type of the subject matches a tab selected among tabs provided per genre of object information (feature (d)).

5.3 In its preliminary opinion, the Board expressed the view that feature A' merely reflected the non-technical user requirement of further obtaining position information indicating the shooting spot of the image. In the face of that requirement, it would be obvious for the skilled person to modify the device of D1 in the way defined by feature A'.

In its written reply and at the oral proceedings, the appellant argued that the provision of a shooting spot was comparable with the technical field of navigation, because the user was guided to a point in space where they could take a picture according to their expectations. Navigation was implicit in the claim. Explicitly claiming the feature of navigation was not necessary for credibly assisting the user in performing the technical task of reaching the point in space; this directly and necessarily followed from using the device claimed and transmitting the position to the terminal device.

Providing improved position information to a user was based on technical considerations. It was based on the presence of physical elements correlated with the position information, which reflected technical characteristics of the real world. In addition, it enabled the user, for instance, to move a vehicle along a direction aimed at reaching the point in space for shooting a photo in the same manner as a navigation system enabled a user to reach a desired destination. Moving had physical consequences and was hence a technical process. Reference was made to decision T 2035/11 of 25 July 2014, Reasons 5.2.1, and to decision T 1670/07 of 11 July 2013, Reasons 13.

5.4 In decision T 2035/11, claim 1 of the main request covered a personal computer executing route-planning software for determining, from a database of roads, the lowest-cost route between a beginning point and a desired destination based upon costs of the road segments. The competent Board considered that the purpose of the algorithm was the mere display of an optimal path to the user for cognitive processing. The user could act on the information, but did not need to (Facts and Submissions, section IX, Reasons 5.1 to 5.1.3).

As was stated in decision T 1670/07, reasons 13, a technical effect could arise from either the provision of data about a technical process, regardless of the presence of the user or its subsequent use, or from the provision of data (including data that on its own is excluded, e.g. produced by means of an algorithm) that was applied directly in a technical process. In the case at hand in decision T 2035/11, the data was produced by means of an algorithm and was not applied directly in a technical process, so neither possibility applied. The optimisation algorithm of claim 1 of the main request was thus not considered to serve a technical purpose (reasons 5.1.3). If an invention resided in the application in a technical process of data produced by an algorithm, the application of the produced data in the technical process should be properly reflected in the claim (reasons 5.1.5).

On the other hand, the Board ruled the following:

"5.2.1 [...] providing real-time route-guidance information to a user in dependence on the user's real-world position is a technical task. It involves an interaction between the user and the navigation system, wherein the navigation system continuously measures the user's position using technical means and, on the basis of these measurements, provides the user with information aimed at enabling the user to manage the technical task of moving a vehicle to a desired destination.

Although the completion of this technical task depends on the user acting upon the provided route-guidance information and hence on an intervention by the user, it does not rely on subjective considerations by the user or on psychological effects. The user may still decide to ignore the route-guidance information, but that does not detract from the technical character of the navigation system as a technical tool to be used interactively in a technical process and not merely in a preparatory phase as a substitution of what could also be done using pencil and paper."

The competent Board in T 2035/11 then concluded that a mathematical route-planning algorithm, when used in a navigation system comprising a position-determining device and route-planning functionality dependent on the actual real-world position of the system, provided a technical contribution at least to the extent that it produced information that enabled the route-guidance functionality. A route-planning algorithm did this by producing a route in the form of an ordered list of road segments based on real-world map data (see reasons 5.2.2).

5.5 In the present case, the rationale of T 2035/11, reasons 5.2.1 and 5.2.2, does not apply. The device of claim 1 does not use information about the user's position. Neither the distinguishing features nor claim 1 concern navigation, route planning or real-time guidance. The position information returned by the device in feature A' can be used for any purpose, for example for merely presenting the information to the user, i.e. for presentation of information as such. Contrary to the appellant's arguments, the claim cannot thus be considered to implicitly specify the navigation.

The text of claim 1 does not reflect the application of the position information in a technical task of guiding the user to a point in space. Therefore, it cannot be said that in the present case providing position information to a user is based on technical considerations. The mere fact that the position information concerns physical elements in the real world is not sufficient for establishing a technical contribution of providing the position information (T 154/04, OJ EPO 2008, 46, reasons 20; T 2035/11, reasons 5.1.4).

The Board is therefore of the opinion that feature A' meets the non-technical user requirement of adapting the device of document D1 to further return position information indicating the shooting spot of the image. At the priority date of the present application, it was common practice to store position information with images taken, e.g. by digital cameras (see also paragraph [0002] of the present application), such position information corresponding to the shooting spot. It would therefore be obvious for the skilled person to store position information with each image and provide, as a result of the search, the position information for the matching images. That is, it would be obvious to modify the device of document D1 in the way defined by feature A'.

5.6 In the grounds of appeal, the appellant argued that features A and B interacted to provide a synergetic technical effect of improving accuracy of the information provided to a user. By including feature B, it was possible to search image data including an object of a matching type and further to estimate a shooting direction based on the "difference of coincidence of arrangement of object in searched image data and difference of position information of [...] each image data".

The Board does not find this argument convincing because the claim does not define how the result set is calculated and in particular does not even mention a shooting direction or specify that the shooting direction is estimated. Besides, feature B does not change the information available for the search, only how it is selected by the user (if interpreted in the light of the description).

It is further questionable whether feature B limits at all the scope of claim 1 directed to an information providing device. The claim does not define whether the tabs and object genres are supported by the information providing device or by the terminal. In any case, tabs are well-known user interface elements. It would be obvious to modify the user interface of document D1 to use tabs instead of container windows, and, on the basis of non-technical requirements, to provide tabs for category genres (corresponding to object genres in claim 1) as well.

5.7 Therefore, the subject-matter of claim 1 of the main request is not inventive (Articles 52(1) and 56 EPC).

First and second auxiliary requests

6. Claim 1 of the first auxiliary request additionally specifies that:

(f) the receiving means is arranged to receive condition information indicating a condition related to the position information; and

(g) the searching means is arranged to perform the comparison of the image data matching position information which satisfies the condition indicated by the received condition information.

7. Claim 1 of the second auxiliary request differs from claim 1 of the first auxiliary request in that it additionally specifies that:

(h) the receiving means is also arranged to receive specified object information of a shooting target; and

(i) the searching means performs comparison of the image data [...] captured within a predetermined range based on position information of the shooting target.

8. Claim interpretation

The device according to claim 1 of the first auxiliary request additionally takes into account, in the search of images containing an object of a given subject type in a given arrangement, a condition on the position of the shooting spot. The device of claim 1 of the second auxiliary request additionally takes into account a condition regarding a shooting target and searches for images captured within a predetermined range of the shooting target. At the oral proceedings, the appellant argued that the shooting target was to be interpreted, as disclosed in paragraph [0060], as a particular subject or place, such as Mount Fuji.

9. Inventive step - claim 1

9.1 The appellant argued with regard to the first and second auxiliary requests that the invention solved the technical problem of efficiently performing a search for image data. The amount of data processed by the device during the search was reduced by processing only data satisfying technical conditions related to the position information. The condition information was intrinsically based on technical considerations because it reflected physical characteristics of the real world.

9.2 The Board does not find these arguments persuasive. Claim 1 of the first auxiliary request merely specifies that a further search condition relating to the position of the shooting spot is used in the search. It does not further describe that search condition or how the search is performed. Therefore, a gain in efficiency cannot be established.

Claim 1 of the second auxiliary request adds a further search condition regarding a shooting target and searching for images captured within a predetermined range of the shooting target. The claim does not further specify what the predetermined range is or how it depends on the position information. The Board is therefore not convinced that the search within the predetermined range is based on technical considerations regarding a more efficient search.

Consequently, the additional features of claim 1 of each of the first and second auxiliary requests reflect non-technical requirements regarding the information of interest for the user. In the Board's view, it would be obvious for the skilled person to support those additional search criteria in the device of document D1 by modifying the receiving and searching means in the way suggested by features (f) to (i).

9.3 The Board further notes that according to paragraph [0002] of the present application, the idea of searching images corresponding to a shooting spot close to an arbitrary location was known from the prior art.

9.4 Therefore, the subject-matter of claim 1 of each of the first and second auxiliary requests does not involve an inventive step (Articles 52(1) and 56 EPC).

Third and fourth auxiliary requests

10. Claim 1 of the third auxiliary request differs from claim 1 of the first auxiliary request in that it additionally specifies that

(j) when a position at which object information is arranged by a user and an arrangement position at which corresponding object information is represented in image data are compared, a range in which a degree of coincidence is decided is gradually expanded.

11. Compared to the second auxiliary request, claim 1 of the fourth auxiliary request further specifies that

(k) the predetermined range is increased stepwise based on the position information of the shooting target.

12. Claim interpretation

Claim 1 of the third auxiliary request and claim 1 of the fourth auxiliary requests additionally specify, respectively, that the condition on the arrangement of the subject in the image is gradually relaxed (by a range) and that the predetermined range from the shooting target is increased stepwise (based on the position information of the shooting target).

13. Inventive step - claim 1

13.1 The appellant argued that with the claimed device it was possible to reduce the likelihood of occurrence of "an error preventing the provision of information in reply to a request". The search for image data was more efficiently performed. Features (j) and (k) should be considered technical because the search requirement did not necessarily (directly) depend on the information requirement of the user and avoided the situation in which, based on the (direct) information requirement of the user, no response was generated by the system.

However, in the claimed invention the images are not searched for a technical purpose and the Board cannot identify any technical motivation in the decision to relax the search conditions. The question of whether the search requirements should be relaxed or not, for instance in order to avoid an empty search result set, depends on non-technical aspects regarding the information requirements of the user, not on technical considerations. Features (j) and (k) result straightforwardly from those non-technical requirements of relaxing the search condition.

13.2 Therefore, the third and fourth auxiliary requests do not satisfy the requirements of Articles 52(1) and 56 EPC.

Fifth auxiliary request

14. Compared to the second auxiliary request, claim 1 of the fifth auxiliary request further specifies that

(l) the predetermined range is determined based on a rate of a size of the arrangement information arranged in the frame with respect to the frame.

15. Claim interpretation

The additional feature specifies that a predetermined range is obtained based on the ratio of the size of the subject arrangement to the frame. For example, the device derives a range from the relationship between the size of the mountain object in the frame and the size of the frame (see the examples in the grounds of appeal and paragraph [0060] of the description). It then searches for images of the shooting target, e.g. Mount Fuji, "captured within [the derived] range based on position information of the shooting target".

16. Inventive step - claim 1

16.1 The appellant argued that the invention according to claim 1 of the fifth auxiliary request made it possible to "determine an extraction target range of image data" as explained in paragraph [0060] and that it led to a reduction of the processing load because the amount of data to be extracted was small. According to the appellant, the amount of data to be extracted depended inversely on the size of the object relative to the frame.

16.2 The Board does not recognise such an effect. Since the result of the search is different for different sizes of the object relative to the frame, the searches are not comparable and, thus, it cannot be said that a reduction of the processing load is achieved. Furthermore, there is no basis in the claim or in the whole application for the appellant's allegation that the amount of data to be extracted depends on the size of the object relative to the frame.

Feature (l) supports, in combination with the other features of claim 1, a search for images showing a specific shooting target in a given size. It is notoriously known, and a non-technical geometric insight, that the size of a subject's projection on an image area depends on the distance of the shooting spot from the subject. Therefore, feature (l) results from obvious and non-technical geometric considerations regarding the projection of the shooting target's image on the image area depending on the camera position.

16.3 Therefore, the subject-matter of claim 1 of the fifth auxiliary request is not inventive (Article 56 EPC).

Conclusion

17. Since none of the requests is allowable, the appeal is to be dismissed.

Order

For these reasons it is decided that:

The appeal is dismissed.