T 0792/01 (Inserted Indicia/PRINCETON) of 9.10.2002

Summary of Facts and Submissions

I. The appeal is against the decision of the Opposition Division revoking European Patent 0 595 808 based on international application WO 93/02524 (PCT/US91/05174) with international filing date 19 July 1991, claiming no priority.

II. The European patent was granted with a claim 1 reading:

"A method of altering a video image display to provide a substituted display of desired indicia within a portion (11, 13) of the video image display on a frame-to-frame basis in conformity with the selected portion on a frame-to-frame basis, the TV camera (14) being operable at various different perspectives and/or magnifications to create the display, the portion (11, 13) corresponding to a target in the scene; the method comprising:

creating said desired indicia by creating one or more predefined insertable images and storing said images in a memory (27);

using an operator interface (16, 21) to identify one or more landmarks in the scene from the image of the scene in the display and storing a representation of the landmarks in memory;

using the operator interface (16, 21) to select the target in the scene from the image of the scene in the display, the selection of the target being distinct from the identification of the landmarks;

using the representation of the landmarks in the memory to recognise the landmarks in the scene using pattern recognition (17), thereby to determine the location of the target, and hence recognise the position of the portion in the display; and

inserting (20) said desired indicia into said display to replace the portion, thereby to synthesise a modified display (30)."

III. The patent was based on a PCT application which as originally filed contained nine apparatus claims and eleven method claims, of which claim 10, the broadest and sole independent method claim, read:

"10. A method for altering a video image to provide a substituted display of desired indicia within a preselected portion of said video image display on a frame-to-frame basis and independent of the size of said selected portion on a frame-to- frame basis which size is a function of the TV camera perspective employed to create said display, comprising the steps of:

selecting said portion of said display to be substituted,

recognizing said selected portion of said display on a frame-to-frame basis and independent of the size of said portion with respect to said display,

generating a video image of said desired indicia, and

inserting said image of said desired indicia within said recognized portion of said display on a frame-to-frame basis."

IV. An opposition was filed against the patent on the grounds under Article 100(a) and (c) EPC. It was argued inter alia that the feature of claim 1 as granted ".. the selection of the target being distinct from the identification of the landmarks.." had no basis in the application as originally filed. Oral proceedings took place at which the patent was defended on the basis of claim 1 as granted.

V. The Opposition Division found that the application as originally filed only disclosed a selection of landmarks which related to and was equivalent to the selection of a target and thus there was no basis for the feature of claim 1 as granted "... the selection of the target being distinct from the identification of the landmarks..." in the application as originally filed. On this basis the Opposition Division revoked the patent.

VI. An appeal was made by the Appellant (proprietor) against this decision by a Notice of Appeal filed on 11 July 2001, with payment of the appeal fee on the same day, asking that the decision under appeal be set aside and the patent be maintained.

Grounds of Appeal were filed on 28 September 2001 by a newly appointed representative, requesting maintenance of the patent on the basis of a new set of claims including both the claims as granted and new independent method claims 25 and 29, and claims 26 to 28 and 30 to 32 respectively dependent thereon.

The Appellant also filed statements by two expert witnesses in support of its argumentation that the claims had a basis in the original application, and a statement by the Chairman of the Directors of the Appellant explaining, by reference also to two other applications by the same inventors as the patent in suit, including GB 9102995.9, what the Appellant had been working on and what it thought it was obtaining protection for in the present patent.

VII. The Respondent filed a response on 21 March 2002 arguing that the appeal was inadmissible as in the Grounds the Appellant by including new claims 25 to 32, was asking for more than the extent that had been indicated in the Notice of Appeal. Further claims 1 to 24 contravened the requirements of Article 123(2) and, having regard to WO-A-91/15921 (D1), 54 EPC, and new claims 25 to 32 contravened the requirements of Article 123(2), (3) EPC.

The Respondent requested that the appeal be refused and asked for an apportionment of costs under Article 104 EPC by reason of the Appellant's failure to submit documents or discuss issues at the appropriate stage earlier in the proceedings.

VIII. In a communication dated 28 May 2002, the Board of Appeal indicated its provisional opinion on inter alia the following points:

- The appeal appeared to meet the formal requirements of Articles 106, 107 and 108 EPC and of Rule 64 and thus to be admissible.

- Rule 57a EPC provided that the description, claims and drawings may be amended, provided that the amendments are occasioned by grounds of opposition specified in Article 100, even if the respective ground of opposition had not been invoked by the opponent. As in the main request the claims as granted all remained unchanged, the addition of claims 25 to 32 could not be considered as meeting any ground of opposition: such addition could, if anything, only give rise to new objections. It appeared that the main request would have to be refused as contravening Rule 57a EPC.

- The relevance of the arguments in the Grounds of Appeal to the Article 123(2) EPC issue was made difficult to appreciate, by being focussed on terms such as "landmark" and "target" which were not used at all in the original text or had no clear definition therein, rather than on the language of the original text.

- Actual evidence by experts stating what they derived from the original application text was not normally helpful, as this was an issue for decision by the Board. The exception would be where a specific term used had a special meaning in the particular art.

IX. On 9 August 2002, the Appellant filed a further submission stating that since it now had arrived at the view that the novelty of claim 1 as granted could not be upheld over D1 for the reasons given by the Respondent in the opposition, substitution of new claims for independent claims 1 and 17 as granted was proposed, with claim 1 reading:

"1. A method of altering a video image display by substituting a desired indicia within a selected portion (11, 13) of the video image display on a frame-to-frame basis, the substituted indicia being in conformity with the selected portion of video image display on a frame-to-frame basis, a TV camera (14) being operable at various different perspectives and/or magnifications to create the display; the method comprising:

(a)creating said desired indicia by creating one or more predefined insertable images and storing said images in a memory (27);

(b)using an operator interface (16, 21) to identify in an image of a scene an outline of an image to be recognised by a pattern recognition program of an image analyser and storing a representation of the outline in memory;

(c)using the operator interface (16, 21) to select the portion of a video display to be substituted, the selection of the portion of the video display to be substituted being distinct from the identification of the outline;

(d)using the representation of the outline in the memory to recognise the outline in an image of the scene using the pattern recognition program (17), thereby to determine the location of the portion of the video display to be substituted; and

(e)inserting (20) said desired indicia into said display to substitute the portion of the video display, thereby to synthesise a modified display (30);

characterized in that the outline of the image corresponds to a visible marking on a sports playing surface and the inserted image appears to form a part of the sports playing surface."

X. Oral proceedings before the Board were held on 9 October 2002.

XI. The Appellant's argumentation can be summarised as follows:

- With the patentee's system computer generated images can be placed anywhere within the region defined by a mathematical model, including deep within featureless areas such as football pitches and tennis courts.

- It was necessary to read the claims with the eyes of the skilled person.

- Having regard to the description and claims as originally filed, the scope of claim 1 had not been extended compared to that of original claim 10 which could be considered to correspond to present claim 1.

- The substitute claim 1 submitted on 9 August 2002 was in two part form, with the entire language of claim 1 as granted in the preamble reflecting its status as a description of the closest prior art D1. The new characterising language was clearly supported by the specific embodiment and the drawings as originally filed.

- In original claim 10 it was made clear that a "portion" was selected which later on was recognized on a frame-to-frame basis and a video image of the indicia was inserted within that recognized portion. Thus original claim 10 identified the outer borders of the scope intended to be protected. However when reading the description with the eyes of the skilled man it was apparent that there existed a second step in that the "indicia" in the terms of original claim 10 must be located somewhere within said "recognized portion".

- It was not necessary to refer to any specific parts of the description. Rather the description as a whole had to be understood in the light of the common general knowledge in the field concerned. Thus it was necessary to put in the words "direct" and "indirect" at appropriate places, as the skilled person would automatically do so when reading the original text of the patent application.

- It was self-evident for a skilled person that the selection of the area to be substituted had to be done in two steps. Thus the skilled person would understand a text, as in the original application, where it was stated that an "area" which was selected within a given video image and later on, when this "area" was recognized and replaced with a desired content, for example a logo, in the way that it was necessary in a first step to identify an outline of the area and in a second step to identify the exact position of the logo in this area. The mention of the selection of an area indirectly meant to the skilled person that an additional step had to be taken to position the logo. The skilled person in the technical field concerned had to be considered as a very clever software and hardware expert, also being familiar with TV-techniques, since the field concerned was complicated and required a very good knowledge in the fields concerned in order to be capable to create new applications which could be marketed.

- In fact, it was self-evident for a skilled person to minimize the number of pixels to be replaced and therefore only the pixels of the logo were replaced and not the pixels between the logo and the identified outline of the selected area. Also because of that, it followed that there had to be two steps, one identifying the borders of the area and a second step for positioning the logo within those borders. In the description moreover the possibility was explicitly mentioned that a logo could be highlighted by moving it during advertising, which also must indicate that the logo could be positioned in different ways within the selected area.

- The affidavits of the three different experts filed all confirmed that a skilled person would interpret the original documents of the application as disclosing two steps of selection as now claimed by the independent claims.

- Enlarged Board Decision G 1/93, in particular point 16, made clear that where a feature merely excluded protection for part of the subject-matter claimed in the original application, the adding of such feature could not reasonably be considered to give any unwarranted advantage to the applicant and therefore could be maintained in the claim without this violating Article 123(2) or(3) EPC. In the present case added feature (c) was such a feature since it only restricted protection.

XII. The Respondent's argumentation can be summarised as follows:

- Throughout the original description and claims it was made clear that only a single selection step was used, nowhere was it hinted at that this could be divided into two substeps. In particular the following parts of the original application text (PCT/WO 93/02524) all suggested that only one step existed:

page 2, line 29 to page 3, line 2;

page 3, lines 21 to 29;

page 8, lines 14 to 27;

page 11 lines 2 to 7;

page 16, line 28 to page 17, line 9;

- The paragraph bridging pages 16 and 17 which in the opposition proceedings were used by the Appellant to defend its case also taught that the selection step is done in one run. Moreover according to the original application, also a back portion of a player's clothing or even a ball, such as a basket ball, could be used for advertising. Also in those cases there was no mention of two steps.

- Document D1 was only prior art for the purposes of Article 54(3) EPC. The preamble of claim 1 could not be based on such a document.

- Costs were requested on the basis of the Appellant having filed on appeal new independent claims 25 and 29 and corresponding dependent claims 26 to 28 and 30 to 32 respectively. Although no longer maintained by the Appellant after the Board's communication the Respondent had nevertheless incurred much additional work by having to study them and argue against them as not meeting the requirements of Rule 57a EPC. Also the late filed evidence on appeal in the form of three long statements by experts required additional study time, and in the circumstances it would be equitable to make an apportionment of costs in favour of the Respondent.

XIII. The Appellant (proprietor) requested that the decision under appeal be set aside and that the matter be remitted to the Opposition Division for further prosecution on the basis of claims 1 and 17 as filed with letter dated 9. August 2002 and claims 2 to 16 and 18 to 24 as granted.

The Respondent (opponent) requested that the appeal be dismissed and that there be an apportionment of costs in his favour in relation to considering new evidence and inadmissible and/or belated requests submitted by the appellant.

XIV. At the end of the oral proceedings the Board announced its decision.

Reasons for the Decision

Admissibility

1. The appeal meets the requirements set out in Rule 65(1) EPC and is therefore admissible. That the set of claims submitted with the Grounds of Appeal does not comply with Rule 57a EPC because of the addition of further claims, does not make the appeal itself inadmissible: an objection on this basis can be dealt with in the course of the appeal.

Article 123(2) EPC

2. Although the Appellant, in response to an objection made by the Board in its communication, has changed the wording of claim 1 so as not to use terms such as "landmark" and "target" which did not appear in the application as originally filed, claim 1 as now requested still defines two distinct steps for selection of the "outline of an image to be recognised" (feature (b) - corresponding to "landmark" in the refused claim) and the selection of the "portion of a video display to be substituted" (feature (c) - corresponding to "target" in the refused claim 1 as granted). The critical question thus remains whether these two features are supported by the original application or whether they represent additional matter in the sense of Article 123(2) EPC.

3. The evidence of the Appellant included a reference to Patent Application GB 9102995.9 by the same two inventors as the patent in suit, to indicate what the Appellant had been working on and what it thought it was obtaining protection for in the present patent. This GB patent application includes two passages at page 1, lines 16 to 20, and page 2 lines 4 to 8 reading respectively:

"The essence of the invention is the method of altering video images by selection all [sic] or part of an object or objects within a 1st video image, then recognizing that same part or all of an object in each or any of a subsequent stream of video images, and using the position of that object or part of it, as reference to accurately insert and position a 2nd still or video image into each or any of the stream of video images."

"The advertising method of this invention requires very precise positioning of a new image into an existing image. This requires pattern recognition of preselected features, such as the goal posts in a scene of a soccer match. These features can then be used to locate the position, size and perspective of an artificial electronic billboard, which is added to the video image and appears to the end user as if it were part of the original scene."

Unfortunately for the Appellant, neither of these passages, which might arguably have provided a basis for the claim 1 now contended for, appeared in the application that led to the patent in suit. Under the European Patent Convention the subject matter that can be protected depends critically on the content of the application as filed. The fact that the applicant may at the time of the application have been in possession of and working on other subject matter is of no relevance if this other subject matter did not appear in the text as originally filed of the application leading to the patent under consideration.

4. The test applied by the Boards of Appeal under Article 123(2) EPC for the purpose of assessing whether or not a patent has been amended in such a way that it contains subject-matter which extends beyond the content of the application as filed is to consider whether everything in the amended patent can be directly and unambiguously derived from the application as filed, in the light of the general knowledge of the person skilled in the relevant art. It is what can be deduced from the text as originally filed that is of critical importance: intentions of the inventors or drafter of the text are irrelevant if the original text does not reflect such intentions.

5. Passages which the Board regards as of significance are the following:

page 2 lines 25 to page 3 line 11:

"... As will be explained, the present application relates to a method of advertising using existing television transmission facilities in which the advertiser selects predetermined areas in a video scene. These areas are then recognized using existing technology as pattern recognition techniques and the content of these areas is replaced by inserting an image or images of the advertisers choosing. In order to accomplish the present invention one must recognize the selected predetermined area and replace the same with advertising indicia in real time. The inserted indicia is then blended into the original video image in such a way that it appears part of the television scene and cannot be otherwise discerned by a typical viewer. The inserted advertising or messages can further be highlighted to the viewer by moving the message, changing color, associating the message with sound or otherwise imparting modulation to the message to distinguish the message during the course of the television program...."

page 3, line 22 to page 4, line 10:

"... one must basically select an object or area within a given video image. For example the area selected may be a televised tennis court. This area is then recognized in each of a subsequent stream of video images and once recognized the video content of that area is replaced with a desired content which may be an advertising commercial, logo or some other matter. The insertion and replacement of the existing video image with the new material is accomplished completely independent of the size of the image in subsequent television scenes or frames. The system described herein will always recognize the particular area selected no matter what the size of the area is with respect to the remaining television picture. This, therefore, creates the illusion to the viewer that the replaced subject matter is actually that subject matter which is being televised in real time. Thus, the methods and apparatus described enable advertisers to add or merge images, messages or slogans to preexisting video images in such a way that they appear to be part of the original image even when the original video stream is of a live event being distributed in real time.

Thus, the techniques described herein are superior to prior art techniques of inserting video information. In order to do and accomplish the results of this particular application an electronic device is required to select, recognize and substitute images to be broadcast on live televised events. The device detects part or all of an object or objects within a video image and uses the position of the object or part thereof as a reference. The device then positions another still or video image into the original image at the reference location. The still or video image is positioned accurately so that the final composite image appears as though it is part of the original scene. That is, the added image is seamlessly and realistically incorporated into the original event. The appearance of the added image will conform to the appearance of the original scene when the scene is moved, panned, magnified, zoomed or otherwise altered in size or perspective."

page 8 lines 14 to 27:

".. Thus as one will ascertain the advertising method and apparatus of this invention requires very precise positioning of a new image into an existing image. The techniques to be described employ pattern recognition apparatus which recognizes pre-selected features or portions of a typical television scene. These features can then be used to locate the position, size and perspective of an artificial electronic advertising message which is added in the exact congruency with the replaced recognized image and appears to the end user as if it were part of the original scene. The added material is inserted into the pre-selected scene independent of size of the selected feature as varying on a frame to frame basis."

page 11 lines 2 to 7:

"..As will be explained in one embodiment of the invention, the fore court 11 as well as the back court 13 or the entire tennis court 12 is subjected to a pattern recognition algorithm whereby an advertisement is actually inserted directly on the portions of the court as shown in Fig. 2. "

page 16 line 27 to page 17 line 15:

"..Operation of the system as indicated is basically as follows. The operator views the image as presented on the monitor television receiver 16 and marks selective portions of the outlines of the court or the net area to choose or direct that portion of the displayed image to be recognized and where the advertising indicia will be inserted. In this particular example the operator will mark the upper court 11 or the lower court 13 by means of a light pen or other suitable device and therefore instruct or command the image analyzer 17 to recognize these markings in subsequent video images of the court 12 as televised by the camera. The image synthesizer is now employed to locate, position and orient, including the correct magnification, the indicia or logo, which is taken from the second image source 27 so that it appears as part of the original scene. The logo which is generated by the second image source 27 is made to appear as part of the background and does not interfere with objects or people in the foreground. This is accomplished by allowing the logo to key over specific colors or ranges of colors such as the color of a court in the tennis match."

6. The overwhelming impression of these passages is that the operator makes only a single selection in the image presented: namely that of the outlines of the portion that is to be replaced. It appears that this portion may be totally replaced by a logo on its own background, or replacement may be accomplished by allowing the logo to key over only specific colours. If the portion selected is totally replaced, the operator will have to determine the relative sizes of the logo and its background that form the replacement, but this is not a selection made in the image of the sporting scene presented, and so cannot provide support for the wording of present claim 1.

7. The Appellant in the oral proceedings before the Board tried to convince the Board that in this case the whole text of the original application should be looked upon through the eyes of the skilled person, this person having the general knowledge in the field of computers and TV techniques, and that when reading the whole text of the original application the skilled person would at the identification of the different method steps automatically put in the words "direct" or "indirect" before the step concerned and therefore inevitably identify a method having two distinct steps as defined in claim 1. This is not an acceptable approach under Article 123(2) EPC. The contents of the application as filed do not include what further ingenious embodiments a skilled reader might contemplate after having studied the application in depth. This is particularly so when questions might arise as to whether the original application gave sufficient information on how to put such further ingenious embodiments into practice. In the present case what is described is direct replacement of an outlined portion in one plane. Replacement not of the originally outlined portion, but of something else calculated by reference to this originally outlined portion would seem to require more complicated calculations, and so is not something which can be treated as directly and unambiguously derivable from the original text.

8. The Appellant in its argumentation also referred to Enlarged Board of Appeal decision G 1/93 (see under point XI above) and argued that feature (c) in claim 1 above was only restricting the scope of the invention and thus excluding protection for part of the subject-matter as claimed in the application as originally filed. Therefore such a feature could not be considered to give any unwarranted advantage to the Appellant. Apparently the Appellant related its argumentation to the second part of point 16 of G 1/93, i.e. the part, corresponding to Headnote 2, which relates to addition of features of a non technical character ("without providing a technical contribution to the subject-matter of the claimed invention"). However, in the present case the feature which has been added to claim 1 (feature (c)), clearly is of technical character, since it has an effect on the way the video display can be altered. The appropriate legal principle is to be found in the first part of point 16 in G 1/93 where it is stated that "If such added feature, although limiting the scope of protection conferred by the patent, has to be considered as providing a technical contribution to the subject-matter of the claimed invention, it would in the view of the Enlarged Board, give an unwarranted advantage to the patentee contrary to the above purpose of Article 123(2) EPC". Thus it is apparent that reliance on decision G 1/93 does not assist the case of the appellant.

9. Nor can the Appellant rely on the claim being justified by reliance on document D1, international application WO91/15921 published only on 17 October 1991. This document is only prior art for the purpose of Article 54(3) EPC, and by Article 56 EPC not to be considered in deciding whether there has been an inventive step. D1 was not referred to in the application as filed, nor did the application as originally filed disclose its subject matter. D1 cannot therefore, as pointed out by the Respondent, serve to provide a basis for the subject matter of the preamble of claim 1.

10. The Board therefore concludes that the subject-matter of claim 1 does not meet the requirements of Article 123(2) EPC.

Costs

11. The Respondent has requested an apportionment of costs, since it feels that the additional claims filed by the Appellant with the grounds of appeal and the evidence in the form of statements by experts caused so much extra work that an apportionment of costs in favour of the Respondent is justified.

12. Article 104(1) EPC states that "Each party to the proceedings shall meet the costs he has incurred unless a decision of a .... Board of Appeal, for reasons of equity, orders, in accordance with the Implementing Regulations, a different apportionment of costs incurred during taking of evidence or in oral proceedings." The Respondent sought to argue during oral proceedings before the Board that the costs incurred in the present case arose during the preparation for oral proceedings and therefore should be considered to be covered by Article 104 EPC.

13. However, it appears to the Board that the actions taken by the Appellant can all be considered as a genuine attempt to further its own interests, and not abusive behaviour which would make it equitable to make an apportionment of costs (cf. T 1171/97 of 17 September 1999, not published in OJ EPO). Therefore the Board considers that there is no reason to order an apportionment of costs.

ORDER

For these reasons it is decided that:

1. The appeal is dismissed.

2. The request for apportionment of costs is refused.