T 0643/00 (Searching image data/CANON) of 16.10.2003

Summary of Facts and Submissions

I. European patent application number 93 300 749.4 was filed by CANON K.K. (JP) for an invention in the field of image processing with priority dates of 3 February and 28 April 1992.

II. The invention, which concerns, in particular, searching hierarchically encoded image data, was the subject of a European search yielding just the following two citations, both in the document category "A":

EP-A-0 392 753 (CANON K.K., published in 1990)

SIGNAL PROCESSING: IMAGE COMMUNICATION, vol. 4, no. 2, April 1992, AMSTERDAM, pages 103 - 111, HAMPEL, H. et al. "Technical features of the JBIG standard for progressive bi-level image compression", chapter 2.2. "Compatible progressive/sequential coding".

III. During examination, reference was also made to "standard textbook knowledge" and, in addition, to US-patent US-A- 5. 159 468 (CANON K.K., published in October 1992), a document which was already cited in the application as originally filed.

IV. Essentially on the basis of the EP-document, the application was then refused by the examining division in oral proceedings for lack of inventive step. In addition, claim amendments requested were rejected by making reference to the provisions of Rule 86(4) EPC.

The decision in writing was posted on 19 January 2000.

V. A notice of appeal, including an order for payment of the appeal fee, was filed by the applicant on 10 March 2000. The filing of the written statement setting out the grounds of appeal followed on 26 May 2000.

In oral proceedings held on 16 October 2003 before the Board, the appellant amended the claims, the independent claims reading as follows:

"1. An image processing apparatus for searching an image for output comprising:

input means (2) for inputting image data; storage means (6) for storing hierarchically encoded image data derived from the input image data in an order of registration;

first decoding means (7) for decoding image data stored in said storage means;

video frame memory (9) for storing image data decoded by said first decoding means, whereby images based on said image data stored in said video frame memory can be displayed on a display means (10); and

control means (13) for controlling the operation of the apparatus;

wherein said input means are adapted to receive non- hierarchically coded image data; and

wherein the apparatus further comprises second decoding means (3) for decoding the non-hierarchically coded input image data to generate first image data of a first resolution, hierarchically encoding means (5) adapted hierarchically to encode said first image data with different resolutions and store the hierarchically encoded image data in said storage means;

said control means being adapted to cause said first decoding means (7) to decode in the order of registration the hierarchically encoded data of a predetermined plural number of images stored in said storage means to generate second image data of a second resolution which is lower than the first resolution, to cause said video frame memory (9) to store the second image data (S23-1) in divided areas thereof for display by said display means, and wherein the apparatus further comprises means (11) for selecting one image from said displayed second image data, and wherein said control means is adapted to clear (S61-1) said video frame memory based on the detection of an instruction (S28-1) for displaying the selected image of the predetermined number of simultaneously displayed images in a third resolution higher than the second resolution, to perform decoding of the hierarchically coded data corresponding to the selected image by the first decoding means (7), and to generate an image of the selected image at the third resolution (S63-1), store the generated image of the third resolution in the video frame memory (9), and cause the display of said third image of the third resolution on said display means (10), and

wherein said control means is adapted to perform selectively the following processes:

(i) in the case where an instruction for displaying the selected and displayed third image of the third resolution in a fourth resolution higher than the third resolution is detected (S65-1), said control means is adapted to clear said video frame memory on the basis of the detected instruction, to perform decoding of the hierarchy coded data corresponding to the selected image by the first decoding means (7), and to generate an image of the selected image at the fourth resolution, to store the generated fourth resolution image in the video frame memory (9), and to generate display of said selected fourth resolution image on the predetermined display means (10);

(ii) in the case where an instruction for again displaying said second image is detected (S67-1), said control means is adapted to clear said video frame memory on the basis of the detected instruction, to perform decoding of the hierarchically coded data corresponding to the predetermined number of images again by the first decoding means (7), and to generate said second image (S23-1), store the generated second image data in the video frame memory (9), and to generate a display of the second image data of the second resolution stored in said video frame memory (9) on the predetermined display means (10); and

(iii) in the case where an instruction is received for outputting a selected image, said control means is adapted to perform decoding of the hierarchically encoded data corresponding to the selected image to generate image data of the first resolution and to write the decoded image of the first resolution into a working memory (8)."

"6. A method of searching for an image, the method comprising:

inputting image data into an image processing apparatus;

storing in storage means hierarchically encoded image data derived from the input image data in an order of registration;

decoding encoded image data stored in said storage means ; and

storing the decoded image data in a video frame memory so that the decoded image data can be displayed on display means, and wherein the input data is non- hierarchically encoded image data and the method further includes decoding the non-hierarchically encoded input image data to generate first image data of a first resolution;

hierarchically encoding said first image data with differing resolutions and storing the hierarchically encoded image data in said storage means;

decoding in the order of registration the hierarchically encoded data of a predetermined plural number of images stored in said storage means to generate second image data of a second resolution which is lower than the first resolution;

storing said second image data in divided areas of said video frame memory;

displaying said second image data;

selecting an image from said displayed second image data;

clearing said video frame memory in response to a command to display the selected image of the predetermined number of simultaneously displayed images in a third resolution higher than the second resolution;

performing decoding of the hierarchically encoded data corresponding to the selected image to generate image data of the third resolution;

storing the image data at the third resolution in said video frame memory, and

displaying the third resolution image data, and wherein when an instruction is received for displaying the selected and displayed image of the third resolution in a fourth resolution higher than the third resolution, clearing said video frame memory;

decoding the hierarchically encoded image data corresponding to the selected image to generate image data of the fourth resolution;

storing the fourth resolution image data in said video frame memory, and displaying the fourth resolution image data, and wherein when an instruction is received for redisplaying said second image data, clearing said video frame memory, decoding the hierarchically encoded image data corresponding to the predetermined plural number of images to generate again said second image data;

storing the regenerated second image data in said video frame memory and redisplaying said second image data, and wherein when an instruction is received for outputting a selected image;

decoding the hierarchically encoded data corresponding to the selected image to generate image data of the first resolution corresponding to the selected image and writing the decoded image of the first resolution into a working memory for output."

VI. According to arguments submitted by the appellant, the invention was to be seen in an improved image searching concept. The simultaneous display of, for example, eight images in a reduced resolution ensured the fast realization of multiple image data and allowed the user to grasp the image content at a glance and to select the searched image for output in a fast and efficient manner. The function of switching between different levels of resolution ensured that the user could subject the selected image to a thorough visual check before printing or using it otherwise.

Document D1 was mainly concerned with an automated method for editing images stored in a hierarchically encoded form in a database; only in passing it mentioned the issue of searching images and even then only as a preliminary stage of an image editing process. Except for the hint that the lowest encoded resolution layer of the images should be used for search purposes, the document did not give any relevant technical details regarding the image search but it treated it as a side aspect of the image editing process. Because of the sparsity of the disclosure, the skilled reader would be confronted with many different possibilities to embody the poor information scattered over this document. The search might be based, for example, on names, numbers, categories, or the like and then be displayed in a thumbnail manner. On the display, the images could be rendered one-by one in sequence, or in random groups having overlapping display regions or not, in a fixed or in varying resolutions. It was therefore not obvious to render, in a first step, a plurality of small images in the order of their registration, and to give the user then the possibility to realise the selected image selectively in at least four different levels of resolution.

VII. The appellant, accordingly, requested that the decision under appeal be set aside and that a patent be granted on the basis of claims 1 to 9 submitted at the oral proceedings of 16 October 2003.

VIII. The Board announced the appeal decision in the oral proceedings on 16 October 2003.

Reasons for the Decision

1. The appeal is admissible. It is also allowable on the merits since the grant of a patent can be envisaged on the basis of the amended claims, which meet the mandatory provisions of the EPC.

Amendments and claim requirements

2. The Board is satisfied that the amended claims comply with the requirements of Article 123(2) EPC as well as with the requirements of Article 84 EPC in respect of support by the description.

Independent claims 1 and 6 include all features which are essential to the conversion of non-hierarchical image data into hierarchically encoded image data and to image processing under user control for searching an image which is the basis of the four embodiments of the invention described with reference to figures 6 to 13, 14. to 18, 19 to 23, and 24, respectively. These embodiments were already subject of the original application.

Dependent claims 2, 5, 7 and 9 have a basis in the first embodiment, dependent claims 3, 4, and 8 in the second embodiment of the invention.

3. Furthermore, the Board considers the amended claims to comply with the provisions of Rule 86(4) EPC.

The Board holds that this provision only applies if in connection with the claims as originally filed lack of unity existed with the amended claims, requiring the payment of a further search fee under Rule 46(1) EPC.

The amendments to the claims, however, only narrowed the scope of the original claims 1 and 5 without introducing any alternative solutions so that the subject-matter of the amended claims cannot be nor include "other inventions" within the meaning of Rule 46(1) EPC.

4. The claim wording is also in compliance with the clarity requirement of Article 84 EPC, i.e. the scope of protection as conferred by the claims can be determined and the invention for which the claims seek protection can be examined for patentability (see, for example, decision T 49/99 - Information modelling/INTERNATIONAL COMPUTERS, not published in OJ EPO, point 12).

Most of the claim wording is plainly clear in this sense. Only few definitions require interpretation: it is prima facie left open whether selecting an image and providing the "commands" and "instructions" as defined in the claims are functions of the image processing apparatus or refer to direct user interaction. However, in the light of the description it becomes clear that the invention, and in particular the method of searching an image as claimed, invoke human interaction for "selecting" a particular image from the displayed image data, for providing (at least as a result of such interaction) a "command" to display the selected image, and for providing the "instructions" for displaying, redisplaying and outputting the respective images.

Although there are no principal objections to be raised against claiming subject-matter involving human interaction, such claims have to be carefully examined with regard to the technical character of such features.

Examination for patentability

5. The examination for patentability can be limited to the subject-matter of claims 1 and 6 since claims 2 to 5 and 7 to 9. are truly dependent claims merely defining particular embodiments of the invention.

6. From the description, column 3, lines 34 to 49 as well as from the arguments forwarded by the appellant in the course of the appeal proceedings it must be concluded that the claimed inventions are the result of the idea to make the searching process easier to the (human) user, who has conventionally to comb the images one by one on the display at a high resolution in order to select a particular image for output. This goal is achieved by arranging a plurality of images in a side-by-side manner at a low resolution and providing for hierarchical display at higher resolutions so that a comprehensive survey as well as a fast check for details are possible.

Requirement of novelty (Article 54(1) and (2) EPC)

7. The publication HAMPEL et al. (see point II, supra) and the document US-A-5 159 468 deal with subject-matters like bi- level image compression, image browsing over communication links, and progressive image transmission, but they do not disclose any method or apparatus for searching images in databases in the sense the present application is concerned with. In the present case therefore these citations are irrelevant in respect of the invention claimed. Hence, the question of whether or not the priority dates of the present application are valid need not be considered.

This leaves document EP-A-0 392 753 as the only relevant piece of prior art.

8. This EP-document discloses an image searching apparatus comprising input means (figure 1: reader 4) for inputting non- hierarchically coded image data (from a photoelectrically read original image, see column 2, lines 26 to 27) and decoding means (reader 4) for decoding the non-hierarchically coded input image data to generate first image data of a first resolution (at the digital output of reader 4). Hierarchically encoding means (encoder 9) are adapted hierarchically to encode said first image data with different resolutions and to store the hierarchically encoded image data in storage means (database 5) in an order of registration. Decoding means (decoder 10) are operational for decoding image data stored in said storage means and video frame memory (either memory 26 or a separate memory in display 3) for storing image data decoded by said first decoding means, whereby based on the data stored in said video frame memory the user may display images on display means (display 3) or output them to another system (see column 4, lines 15 to 18). The apparatus is controlled by control means (CPU 1) and comprises means (keyboard 2) by which images can be selected from the image data stored in the database 5 (see column 2, lines 47 to 53).

9. The prior art of document EP-A-0 392 753 seems to aim at a high speed search of images by using the lowest resolution image data of the images hierarchically encoded and stored in the database 5 (see figure 1, and in particular column 1, lines 28 to 34, column 2, lines 51 to 53, column 3, lines 25 to 27, column 4, lines 32 to 34, column 5, lines 4 to 15, column 6, lines 42 to 44 and claim 13). The details how the search could be done, however, remain in the dark.

Apparatus

10. The image processing apparatus of claim 1, therefore, differs from the prior art of document EP-A-0 392 753 in the functionality which allows the image processing apparatus, in response to user input, to decode and render a predetermined plural number of hierarchically encoded images, in the order of registration and at the lowest level of resolution, in separated portions of the display, and to select and render one of these images at any of four different levels of resolution, and finally to output the selected image or to continue the search with the predetermined plural number of images next in the order of registration. The apparatus claimed is thus not anticipated by the EP-document.

Method

11. The method of claim 6 is closely related to the image processing apparatus of claim 1 in that the image search is defined essentially by the steps of operating the said apparatus functions rendering the apparatus novel.

12. In summary it follows that the requirement of novelty is met by the apparatus as well as the method as claimed.

Inventive step

13. The Board applies, as a test for inventive step, the "problem-and-solution approach" according to which an invention is to be understood as a technical solution to a technical problem, assessed by the person skilled in the relevant field of technology in the light of the prior art (see the decision T 641/00 - Two identities/COMVIK, OJ EPO 2003, 352, point 5 of the Reasons).

Apparatus

14. Starting from document EP-A-0 392 753, the objective technical problem solved may be seen in providing a technical tool for efficient search, retrieval and evaluation of images stored in an image processing apparatus.

15. Although in the present case human activities are involved in solving this problem, which activities may relate to the information contents of images looked for and may be motivated by personal interests and/or other non-technical preferences, the Board considers the claimed features relating to the format of images displayed, not to be presentation of information as such.

16. It is true that non-technical aspects may be found in the design and the use of an interface through which the user interacts with a system (see decision T 244/00-Remote- control/MATSUSHITA, not published in OJ EPO). Indeed, presenting information through a user interface, if the only relevant effect of the presentation relates to the visually attractive nature of the graphic design or artwork, does not have technical character. However, in its decision the Board has not excluded that an arrangement of menu items (or images) on a screen may be determined by technical considerations. Such considerations may aim at enabling the user to manage a technical task, such as searching and retrieving images stored in an image processing apparatus, in a more efficient or faster manner, even if an evaluation by the user on a mental level is involved. Although such evaluation per se does not fall within the meaning of "invention" pursuant to Article 52 EPC, the mere fact that mental activities are involved does not necessarily qualify subject matter as non-technical since any technical solutions in the end aim at providing tools which serve, assist or replace human activities of different kinds, including mental ones.

Furthermore, in accordance with case law, the use of a piece of information in a technical system, or its usability for this purpose, may confer a technical character on the information itself in that it reflects the properties of the technical system, for instance by being specifically formatted or processed (see decision T 1177/97-Translating natural languages/ SYSTRAN, not published in OJ EPO, point 2 of the Reasons). Additionally, functional data (line numbers, coded picture lines, addresses and synchronisations) recorded on a record carrier to be used in a picture retrieval system are to be distinguished from the cognitive content encoded (see decision T 1194/97-Data structure product/PHILIPS, OJ EPO 2000, 525, point 3.3 of the Reasons). Even if the overall information could be interpreted in an infinite number of different ways in other technical or human contexts, this does not detract from its technical function in the relevant context of the claimed invention (see T 1194/97, at the end of point 3.3). In this context, the Board cannot see a fundamental difference between an image processing system using a specific electrical picture access structure (binary code string) for easy image retrieval and an image processing system using a specific optical picture access structure (simultaneous display of a plurality of low resolution images) for the same purpose. Whether the actual retrieval and selection process is then carried out automatically or with the aid of human interaction is, in the Board's view, not decisive for the technical character of the claimed image processing functions.

17. In the present case, the functions/steps of processing the images in a specific format, i.e. a predetermined plural number of images in a side-by-side manner at a low level of resolution, and allowing selection and display of an image at higher resolutions provide information to the user in the form of a technical tool for an intellectual task he has to master, and hence contribute to the technical solution of the technical problem of an efficient search, retrieval and evaluation of images stored in an image processing apparatus insofar as this is supported by the specific image format chosen. In this respect it is neither mere aspects of art design, like a pleasant look or feeling, that matter, nor the mere information content of images presented, but the organisation of an overall image structure in view of a technical problem.

18. The invention solves this problem by providing a bundle of image processing functions which, if applied in a concerted manner by the user, ensure a time-saving and well manageable search operation as compared to the rudimentary disclosure of document EP-A-0 392 753. Even if each single one of these image processing functions, taken alone, were considered obvious to the skilled person in the light of common general knowledge and normal programming practices, in the technical context of an image database storing hierarchically encoded image data, these functions combine to yield a fast, easy to handle and in summary efficient searching tool, which goes beyond the mere aggregation of normal design options. The fact that the prior art lacks any clear hint to the claimed combination of image processing functions thus carries weight and must lead to the conclusion that the presence of inventive step as required by Article 56 EPC has to be acknowledged.

Method

19. Since the method of claim 6 involves the application of the said bundle of apparatus functions the requirement of inventive step is met by the method as well, substantially for the same reasons as indicated above in respect of the apparatus claim.

Summary

20. It follows that the amended claims form a valid basis for granting a patent. The adaptation of the description to the claims can be the subject of the further prosecution of the application in the first instance.

ORDER

For these reasons it is decided that:

1. The decision under appeal is set aside.

2. The case is remitted to the examining division with the order to grant a patent with claims 1 to 9 filed at the oral proceedings and a description to be adapted.