T 2156/17 (Detecting suspicious activity using video analysis / NCR Corporation) of 10.6.2022

Summary of Facts and Submissions

I. The present decision relates to the applicant's appeal of the Examining Division's decision to refuse its application for a European patent.

II. The Examining Division held that claim 1 of the main request and of auxiliary requests 1, 2, and 3 did not define inventive subject-matter.

III. As examples of prior art, relevant to the claimed subject-matter, reference was made to documents

D2: EP-A-672 993, and

D6: WO-A-95/29470.

D6 was cited as evidence that correlation of two data items by recurring time stamping was known. D2 was cited as an example of a method for automatically identifying different objects for purposes of customer billing and inventory control.

IV. Concretely, the Examining Division held that the contribution to the prior art, consisting of including the number of products registered and sold as a criterion for identifying transactions as fraudulent, was essentially administrative and thus non-technical. The detection of suspicious activity in the context of department stores or retail establishments was regarded as a normal business activity. Apart from the reference to a "video camera" and "transaction terminal" and to the steps of "automatically analysing frames of video" and of "automatically comparing", nothing in claim 1 of the main request implied the use of any technical feature for carrying out the claimed method.

Rather, the claims defined generic specifications for the skilled person, and their implementation did not require any inventive skill. This appeared all the more true considering that the automation of functions previously performed by human operators was in line with the general trend in technology, as acknowledged, for example, in T 775/90 and T 1175/02.

V. In the statement of grounds of appeal, the appellant requested that the Examining Division's decision be set aside and that a patent be granted on the basis of a main request or one of a first to fourth auxiliary request filed with the statement.

Claim 1 of each of the main request and the first to third auxiliary requests was amended, as compared with claim 1 of each of the correspondingly labelled request underlying the impugned decision, to specify that the methods were computer-implemented.

The fourth auxiliary request was new on appeal. It also defined a computer-implemented method and included the step of "identifying and transmitting to a reviewer for review a segment of the video data where the video event that is not identified as a transaction event exists."

VI. In a communication under Article 15(1) RPBA, the appellant was informed of the Board's preliminary opinion. The Board was minded to admit the main request and the first to fourth auxiliary requests because, for the main request and the first to third auxiliary requests, the amendments did not materially affect the subject-matter and because the fourth auxiliary request addressed the Examining Division's misgivings regarding the invention's technical contribution. By limiting transmission to relevant sequences only, a reduction of data transmission or data storage was achieved, as compared to the methods defined in the previous requests. On the substance, the Board held that the Examining Division had been correct to conclude that the then main request and auxiliary requests 1 to 3 lacked inventive step in view of D2 and D6. Document D1 (US-A-5 083 638) was also a possible starting point for the assessment of inventive step. Document D4 (EP-A-843 293) was also relevant. Auxiliary request 4 lacked inventive step because limiting a transmission to relevant data was an obvious solution, if too much data were being transmitted or stored.

VII. In reply to the Board's preliminary opinion, the appellant filed a new main request and new first and fourth auxiliary requests. These were amended versions of the main request and the first and fourth auxiliary requests filed with the statement of grounds of appeal. The second and third auxiliary requests were maintained as before. If the amendments were considered admissible, the new requests were to replace the corresponding requests on file.

VIII. Claim 1 of each of the new requests additionally comprises, as compared to the correspondingly labelled request on file, the features of one of more of the dependent claims:

New main request: claims 18, 19, and 20.

New first auxiliary request: claim 12.

New fourth auxiliary request: claim 5.

IX. At oral proceedings before the Board, the appellant first clarified that its claim requests were, in order:

- the new main request;

- the main request;

- the new first auxiliary request;

- the first auxiliary request;

- the second auxiliary request;

- the third auxiliary request;

- the new fourth auxiliary request;

- the fourth auxiliary request.

The Board decided to admit the new main request, but not to admit the new first and fourth auxiliary requests. The appellant then withdrew its main request and new first auxiliary request, and filed auxiliary request 1a. Subsequently, the appellant also filed auxiliary request 5, so that at the end of oral proceedings, the following claim requests remained, in order:

I) the new main request;

II) auxiliary request 1a;

III) the first auxiliary request;

IV) the second auxiliary request;

V) the third auxiliary request;

VI) the new fourth auxiliary request;

VII) the fourth auxiliary request;

VIII) auxiliary request 5.

X. With regard to the admissibility of the new main request, and the new fourth auxiliary request, the appellant emphasised that the amendments were meant to address the new objection of lack of an inventive step starting from document D1 that had been raised for the first time in the Board's preliminary opinion. In the appellant's view, a decision not to admit these new requests would de facto deprive it of its right to be heard, under Article 113 EPC, on this new objection.

XI. Auxiliary request 1a was ranked higher than the first auxiliary request, and was meant to replace the latter if considered admissible. The appellant stressed that the filing of auxiliary request 1a was justified by the approach followed by the Board, relying on the criterion of convergence, in not admitting the new first and fourth auxiliary requests. Claim 1 of auxiliary request 1a combined the features of claim 1 according to the main request and claim 1 of the withdrawn new first auxiliary request. It was thus convergent.

XII. Claim 1 of auxiliary request 5 comprises additional limitations, compared with claim 1 of the new main request. The appellant underlined that it had faced objections of lack of inventive step that relied on common general knowledge, for which no evidence had been provided. This had come as a surprise. The filing of auxiliary request 5 was a reaction to this unusual situation and justified its consideration, even at this late stage of the appeal proceedings.

XIII. The appellant emphasised that document D6 was only focused on a manual approach and was not concerned with video analysis. It did not contain any indication as to how analysis of video frames could be performed. It was further underlined that the present invention claimed priority from 2005, when the possibilities in the field of video processing were limited, compared to what is possible now. This made it even less likely that the skilled person would have recognised the benefits of automating the process of D6. The appellant neither agreed nor disagreed with the Board's view that gain control was a well-known process in the field of image processing, but stressed that the recited processing steps in claim 1 of the new main request had to be read in conjunction. The invention was intended accurately to align the transaction data and the video data in order to identify mismatches, while avoiding detrimental effects resulting from autogain correction.

Similar considerations applied to the additional processing steps defined in claim 1 of the auxiliary requests. In the absence of evidence of the common general knowledge in 2005, the skilled person had no inducement to consider automating the process of D6 along the lines of the claimed inventions.

XIV. Claim 1 of the new main request reads:

A computer-implemented method for detecting a transaction outcome, the method comprising:

obtaining video data associated with a transaction area;

analyzing at least a portion of the video data to obtain at least one video parameter concerning at least a portion of a transaction associated with the transaction area, wherein the step of analyzing comprises automatically analyzing frames of video from at least one region of interest in the at least a portion of the video data to identify a respective event indicating the presence of an item associated with the transaction;

obtaining at least one transaction parameter originated from a transaction terminal associated with the transaction area, wherein the step of obtaining comprises analyzing at least a portion of transaction data from a transaction terminal to identify a sequence of items identified as being transacted in the transaction; and

automatically comparing the at least one video parameter to the at least one transaction parameter to identify a transaction outcome, wherein the step of automatically comparing comprises analyzing the sequence of items identified as being transacted in the transaction from the transaction data in comparison to the events produced from analysis of the video data to determine if the item represented by at least one event produced from the analysis of the video data is indicated as an item for transacting in the at least a portion of transaction data, and correlating video timestamps of events from the analysis of the video data to transaction timestamps of items reflected as having been transacted in the transaction data to identify events indicating an item in the video data that does not have a corresponding record in the transaction data,

obtaining video data further comprising:

obtaining video data originating from at least one elevated video camera that monitors a transaction area defining the region of interest; and

wherein analyzing at least a portion of the video data comprises:

analyzing the video data to track items involved in the transaction in the transaction area; and

wherein automatically comparing the at least one video parameter to the at least one transaction parameter to identify a transaction outcome comprises:

comparing the video analysis of the tracked items to transaction data produced from a transaction terminal to identify suspicious activity, the comparison including associating scanned item data with video data such that the timestamp of the scan is synchronized with the timestamp of the video;

wherein analyzing at least a portion of the video data comprises:

defining at least one region of interest within the video data;

calculating an object map that identifies a change between a current image of the at least one region of interest and an updated background image of the at least one region of interest;

isolating at least one operator object within the at least one region of interest;

detecting if an analysis of the object map and the operator object identifies at least one of the removal and introduction of an object to the at least one region of interest; and

applying image adjustment processing to the video data to compensate for an autogain characteristic across an entire area of the video data, the image adjustment processing restricting its application to a portion of the region of interest captured within the video data;

wherein applying image adjustment processing comprises:

restricting focus to objects in the transaction area that remain stationary and that exhibit at least one of a color shift and a brightness shift.

XV. Claim 1 according to auxiliary request 1a differs from that of the new main request in that it includes, at the end of the claim, the following additional limitations:

...;

producing a set of detection events indicating detection of items by at least one detector within at least one region of interest of at least one portion of the video data;

wherein automatically comparing the at least one video parameter to the at least one transaction parameter comprises:

for each detector, comparing the set of detection events for that detector to at least a portion of transaction data taken at the time of the event to identify at least one apparent discrepancy in a number of items detected by that detector from a number of items indicated in the at least a portion of the transaction data; and

identifying an overall suspicion level for the transaction based on apparent discrepancies identified by the at least one detector,

wherein analyzing at least a portion of the video data comprises:

identifying motion of an operator within a region of interest in the transaction area that indicates the presence of an item for transacting within the region of interest;

wherein obtaining at least one transaction parameter comprises:

indicating if a record of the item occurs within the transaction data corresponding to identifying the motion of the operator; and

wherein automatically comparing the at least one video parameter to the at least one transaction parameter comprises:

identifying a situation when the motion of the operator within the region of interest in the transaction area indicates the presence of the item for transacting but the record of the item does not occur in the transaction data, and in response, indicating suspicious activity.

XVI. Claim 1 of the first auxiliary request differs from claim 1 of the new main request (and from claim 1 of the withdrawn main request) in that the whole section starting with the step of comparing the video analysis of the tracked items to transaction data produced from a transaction terminal to identify suspicious activity, until the end of the claim, instead reads:

comparing the video analysis of the tracked items to transaction data produced from a transaction terminal to identify suspicious activity,

producing a set of detection events indicating detection of items by at least one detector within at least one region of interest of at least one portion of the video data;

wherein automatically comparing the at least one video parameter to the at least one transaction parameter comprises:

for each detector, comparing the set of detection events for that detector to at least a portion of transaction data taken at the time of the event to identify at least one apparent discrepancy in a number of items detected by that detector from a number of items indicated in the at least a portion of the transaction data; and

identifying an overall suspicion level for the transaction based on apparent discrepancies identified by the at least one detector.

XVII. Claim 1 of the second auxiliary request differs from claim 1 of the new main request (and claim 1 of the withdrawn main request) in that the whole section starting with the step of comparing the video analysis of the tracked items to transaction data produced from a transaction terminal to identify suspicious activity, until the end of the claim, instead reads:

comparing the video analysis of the tracked items to transaction data produced from a transaction terminal to identify suspicious activity by comparing a first frame and an empty base image by subtraction and

thresholding to create a binary object map of items for containing new objects in the image that are not part of the empty base image, including an operator object and other objects within the region of interest.

XVIII. Claim 1 of the third auxiliary request differs from claim 1 of the new main request (and claim 1 of the withdrawn main request) in that the whole section starting with the step of comparing the video analysis of the tracked items to transaction data produced from a transaction terminal to identify suspicious activity, until the end of the claim, instead reads:

comparing the video analysis of the tracked items to transaction data produced from a transaction terminal to identify suspicious activity,

defining a region of interest within the transaction area;

automatically identifying a first frame of video data that indicates a first set of items in the region of interest;

automatically identifying a second frame of video data that indicates a second set of items in the region of interest, the first set of items being visually distinct from the second set of items; and

automatically indicating the visual distinctness of the first set of items from the second set of items as an event indicating an item existed within the region of interest of the video data; and

wherein obtaining at least one transaction parameter comprises:

obtaining transaction data associated with the transaction terminal associated with the transaction area, the transaction data indicating if the item was registered as a transaction item with the transaction terminal; and

wherein automatically comparing the at least one video parameter to the at least one transaction parameter comprises:

identifying suspicious activity when the transaction data is missing transaction data for an item for which an event indicates the item existed within the region of interest.

XIX. Claim 1 of the new fourth auxiliary request differs from claim 1 of the new main request (and claim 1 of the withdrawn main request) in that the section starting with

obtaining video data further comprising:

obtaining video data originating from at least one elevated video camera ...

which extends until the end of the claim, instead reads:

..., and

identifying and transmitting to a reviewer for review a segment of the video data where the video event that is not identified as a transaction event exists;

wherein analyzing at least a portion of the video data comprises:

applying an automated machine-based video analysis technique to detect a video event of an item associated with the transaction area;

wherein obtaining at least one transaction parameter concerning a transaction that occurs in the transaction area comprises:

identifying transaction data indicating transaction events of items associated with the transaction area; and

wherein automatically comparing the at least one video parameter to the at least one transaction parameter comprises:

determining if the video event is not identified as a transaction event in the transaction data, and in response, identifying a specific segment of the video data that indicates where the video event not identified as a transaction event exists.

XX. Claim 1 of the fourth auxiliary request differs from claim 1 of the new main request (and claim 1 of the withdrawn main request) in that the whole section starting with the step of

obtaining video data further comprising: ...

and extending until the end of the claim, instead reads:

identifying and transmitting to a reviewer for review a segment of the video data where the video event that is not identified as a transaction event exists.

XXI. Claim 1 of auxiliary request 5 differs from claim 1 of the new main request in that the last feature

... restricting focus to objects in the transaction area that remain stationary and that exhibit at least one of a color shift and a brightness shift

instead reads:

restricting focus to objects in the transaction area that remain stationary and that exhibits a color shift,

wherein restricting focus to objects in the transaction area that remain stationary and that exhibit at least one of a color shift and a brightness shift is achieved by:

comparing a current image via subtraction and thresholding from a current updated background image to get a noisy binary map of background pixels,

computing color statistics for the current image and the background image using the background pixels only, wherein the background pixels are used as they correspond to the same, static objects that exhibit the colour shift that needs to be corrected for, and

once the color statistics for the background pixels has been computed, applying a change to the current image to bring its colors into correspondence with the colors of the background image.

Reasons for the Decision

New main request - Admissibility (Article 13(2) RPBA 2020)

1. The new main request was filed after notification of the summons to oral proceedings, in reaction to the Board's preliminary opinion. It addresses, specifically, the objection of lack of inventive step on the basis of document D1, raised by the Board for the first time. Its admission depends on the conditions set out in Article 13(2) RPBA 2020, which reads:

Any amendment to a party's appeal case made [...] after notification of a summons to oral proceedings shall, in principle, not be taken into account unless there are exceptional circumstances, which have been justified with cogent reasons by the party concerned.

2. The appellant's reasons are cogent. Although the objection of lack of inventive step based on D6, relied upon by the Examining Division, was essentially endorsed by the Board in its preliminary opinion, the new objection, based on D1, relied on a completely new approach towards inventive step.

3. The Board considers that the new objection constitutes exceptional circumstances in the sense of Article 13(2) RPBA 2020. It considers, further, that the amendments define additional technical limitations over D1, thus directly addressing the new objection.

4. The new main request is, therefore, admitted into the proceedings.

New main request - Inventive step (Article 56 EPC)

5. Document D6 was cited by the Examining Division as evidence for a method relying on time stamps to establish correlations between two sets of data items.

6. D6 is particularly relevant to the inventive merits of the claimed invention. It discloses a surveillance method for reviewing behavioural and transaction events which occur at one or more point-of-sale (POS) terminals in a retail environment. It comprises the steps of generating video image and transaction signals, generating a synchronising signal, and correlating these three signals (page 5, lines 12-19). It can be used to monitor or identify suspicious activity (page 10, lines 1-4; page 25, lines 7-9).

7. D6 is not limited just to a method of time stamping a sequence of events synchronously recorded on a video tape and a floppy disk, in order, later, to allow video data to be aligned with transaction data.

8. In the terms of claim 1, D6 discloses

A [deleted: computer-implemented] method for detecting a transaction outcome, the method comprising:

obtaining video data associated with a transaction area;

analyzing at least a portion of the video data to obtain at least one video parameter concerning at least a portion of a transaction associated with the transaction area, [deleted: wherein the step of analyzing comprises automatically analyzing frames of video from at least one region of interest in the at least a portion of the video data to identify a respective event indicating the presence of an item associated with the transaction];

obtaining at least one transaction parameter originated from a transaction terminal associated with the transaction area, wherein the step of obtaining comprises analyzing at least a portion of transaction data from a transaction terminal to identify a sequence of items identified as being transacted in the transaction; and

[deleted: automatically] comparing the at least one video parameter to the at least one transaction parameter to identify a transaction outcome, wherein the step of [deleted: automatically] comparing comprises analyzing the sequence of items identified as being transacted in the transaction from the transaction data in comparison to the events produced from analysis of the video data to determine if the item represented by at least one event produced from the analysis of the video data is indicated as an item for transacting in the at least a portion of transaction data, and correlating video timestamps of events from the analysis of the video data to transaction timestamps of items reflected as having been transacted in the transaction data to identify events indicating an item in the video data that does not have a corresponding record in the transaction data,

obtaining video data further comprising:

obtaining video data originating from at least one [deleted: elevated] video camera that monitors a transaction area defining the region of interest; and

wherein analyzing at least a portion of the video data comprises:

analyzing the video data to track items involved in the transaction in the transaction area; and

wherein [deleted: automatically] comparing the at least one video parameter to the at least one transaction parameter to identify a transaction outcome comprises:

comparing the video analysis of the tracked items to transaction data produced from a transaction terminal to identify suspicious activity, the comparison including associating scanned item data with video data such that the timestamp of the scan is synchronized with the timestamp of the video.

9. Document D6 describes a manual process of comparing the video data with the transaction data, and it does not disclose the features added to claim 1 in response to the Board's provisional opinion, according to which analyzing at least a portion of the video data comprises:

defining at least one region of interest within the video data;

calculating an object map that identifies a change between a current image of the at least one region of interest and an updated background image of the at least one region of interest;

isolating at least one operator object within the at least one region of interest;

detecting if an analysis of the object map and the operator object identifies at least one of the removal and introduction of an object to the at least one region of interest; and

applying image adjustment processing to the video data to compensate for an autogain characteristic across an entire area of the video data, the image adjustment processing restricting its application to a portion of the region of interest captured within the video data;

wherein applying image adjustment processing comprises:

restricting focus to objects in the transaction area that remain stationary and that exhibit at least one of a color shift and a brightness shift.

10. The analysing steps define, in relation with the recited steps of "automatically analysing frames of video" and of "automatically comparing", further technical contributions of the claimed subject-matter over D6.

11. The appellant's view, that D6 neither disclosed the step of comparing multiple records of the transaction area at the transaction time nor the step of identifying items found in the video data but not in the transaction data, is rejected. The disclosure of D6 concerns surveillance systems that record transaction events for review at a later date (page 1, lines 4, 5). The very purpose of the system disclosed in D6 is to allow such review of video data and corresponding transaction data by an operator (claim 1). The fact that the comparison is made by the operator, when reviewing the synchronised video and transaction data, does not affect this.

12. The distinguishing features identified above are the consequences of significant differences between the claimed method and the disclosure of D6. In effect, the claimed method differs from the method disclosed in D6, essentially, in that it is computer implemented and performed automatically, and in that the video data originates from an elevated video camera.

13. The selection of an elevated camera merely reflects the need for a complete and unhindered view of the regions to be monitored. By itself, it does not justify the existence of an inventive step. The skilled person would have selected the most adequate location for the camera, depending on the circumstances and the scene to be monitored.

14. The main problem addressed by the distinguishing features identified above is the adaptation of the system of D6 to make it autonomous, so that it does not require the presence of a human operator. Solving this main problem entails addressing further technical issues resulting from the need to exclude false interpretations of video data, as may result from a POS-operator being present in the region of interest or from autogain corrections.

15. As observed by the Examining Division, with reference to decisions T 775/90 and T 1175/02, the mere automation of functions previously performed by human operators (supervisor) is in line with the general trend in technology (Decision, paragraph 19). Although the decision to automate the process of D6 undoubtedly requires a very different approach and involves a multiplicity of secondary issues as to the processing strategy best adapted to the intended purpose, there is no obstacle, in principle, to such a step.

16. In D6, the sets of video and transaction data are both recorded and time stamped. This permits a precise alignment of the two, when replayed afterwards by the respective playback means. Contrary to the appellant's view, no improvement in terms of alignment of video data and transaction data result from the claimed process compared to the manual process of D6.

17. However, the steps of focusing on a stationary area, in order to compensate for the autogain correction, contribute to the reliability of the automated process, insofar as the determination of (moving) transaction items is concerned.

18. In the Board's judgment, the initial decision to automate the process of D6, to make it fully autonomous, would have inevitably led the skilled person to consider subordinated issues associated with the treatment of digital video data. The skilled person would, in particular, have been confronted with the problem of discriminating between (moving) transaction items, a moving operator and (stationary) background objects.

19. The Board has no doubt that comparison by way of subtraction within a sequence of images, in order to identify changes within a region of interest, was well-known at the priority date (2005). The same applies to autogain correction techniques. The Board has also no doubt that the techniques consisting of modifying an image on the basis of statistics obtained from a selected area of it, was also well known.

20. This view is confirmed by the very content of the application, that does not elaborate on these specific aspects of the invention, but refers to them merely in passing. The Board observes that a different finding would have led, in the absence of specific indications as to the content of these measures, to an objection of lack of sufficiency under Article 83 EPC.

21. Nothing that was not obvious results from the step of comparing images by way of subtraction, in order to identify changes, that is, motion within the field of view. Being aware of autogain correction techniques, the skilled person would have recognised that it would directly interfere with the reliability of the comparison process. Faced with wrong interpretations resulting from autogain corrections when a large object, not part of the transaction, appears within the field of view of the camera (page 27, lines 30-35), the skilled person would have considered compensating for this detrimental effect.

22. The solution entails quantifying the effect to be compensated. It implies determining on a statistical basis (histogram analysis) how pixels in a a portion of the image that remained stationary have been affected. Concretely, the portion of the image to be considered was a portion of the image the content of which, in the absence of correction, would not have been modified. This reflects the necessity not to incorporate in the statistical analysis required for the compensation the moving object at the origin of the problem (application, page 27, line 36 - page 28, line 9).

23. The claimed solution did not require any inventive skills. It resulted in a straightforward manner from the application of the known techniques referred to above, and from mere logical considerations as to the use of these well known techniques to solve predictable problems.

24. The subject-matter of claim 1 according to the new main request is thus not inventive in the sense of Article 56 EPC.

Auxiliary request 1a - Admissibility

25. Auxiliary request 1a was filed during the oral proceedings before the Board, so its admission is subject to Article 13(2) RPBA 2020 (cf. point 1, above), and also the criteria of Article 13(1) RPBA 2020.

26. Auxiliary request 1a is an attempt to parry the Board's decision not to admit the new first and fourth auxiliary requests on the ground of lack of convergence, when compared to the new main request. (The admissibility of the fourth auxiliary request is discussed, below).

27. The appellant's arguments are not sufficient to decide favourably on the admissibility of auxiliary request 1a. The circumstances referred to by the appellant are not exceptional, but relate, on the contrary, to ordinary aspects of the appeal proceedings.

28. There is nothing exceptional in the board applying the criterion of convergence to question the admissibility of an auxiliary request freshly filed in appeal proceedings (Article 13(2) RPBA 2020). Although convergence, as stressed by the appellant, is not the sole criterion, there is also nothing surprising in the Board applying it to the new first and fourth auxiliary requests filed in response to the communication of the Board. It was up to the appellant to consider the rules of procedure and the criteria applied according to the case law, and the possible consequences resulting from this new request being filed in reaction to the Board's preliminary opinion.

29. The right to be heard under Article 113 EPC does not entail the right to be given further possibilities to file new requests if earlier attempts fail. In the Board's judgment, the appellant's right to be heard is met under the circumstances, since they were aware of the objections to the admissibility of the earlier requests, and had the opportunity to contest this and present their arguments (T 732/21).

30. Auxiliary request 1a is thus not admitted into the proceedings (Article 13(2) RPBA 2020).

First auxiliary request - Inventive step (Article 56 EPC)

31. The claimed invention is a mixture of technical and non-technical features. The approach generally followed to assess inventive step of the subject-matter of a claim comprising a mixture of technical and non-technical features derives from T 641/00, Two identities/COMVIQ, OJ EPO 2003, 352. This approach was generally endorsed by the Enlarged Board of Appeal in G 1/19 Pedestrial simulation, OJ EPO 2021, A77, points 31-34, and points 35 and 36 with explicit reference to G 3/08, Programs for computers, OJ EPO 2011, point 10.13.2).

32. The distinguishing feature of obtaining video data originating from at least one elevated video camera that monitors a transaction is not inventive for the reasons explained above with regard to the new main request.

33. The claimed method is further distinguished from the method of D6 in that the process is automatic, in that it incorporates a step of comparing the set of transaction events with the transaction data in order to identify a discrepancy in the respective number of items, and in the further step of identifying an overall "suspicion level" for the transaction.

34. Contrary to the appellant's view, the step of comparing the set of detection events with the set of transaction data to identify a discrepancy is not technical. Independently of the nature of the events, it must be stressed that the comparison carried out simply compares data. All in all, the claimed step of comparing said data amounts to a mere comparison of lists by computer means.

35. The origin of the data are without bearing on the claimed step of comparing. The technicality of the transaction, as it manifests itself in the movement of items in a sequence of video frames, is lost when it comes to the step of comparing the data. The Board further rejects the view that the count that results from the comparison is technical in the context of the invention. The mere fact that the identification of a discrepancy is used to generate a "suspicion level" is also not sufficient to confer technical character to said method. The notion of "suspicion" is essentially subjective and as such not technical. The same applies to any parameters that may be derived therefrom.

36. It follows that the only feature of a technical nature, beyond the step of obtaining video data by means of an elevated camera, that distinguishes the claimed method from that of D6 resides in the automation of the process.

37. As explained above, this distinguishing feature reflects a general trend in technology and is not sufficient for an inventive step.

38. The subject-matter of claim 1 according to the first auxiliary request is thus not inventive in the sense of Article 56 EPC.

Second auxiliary request - Inventive step (Article 56 EPC)

39. Claim 1 of the second auxiliary request includes limitations regarding the step of comparing the video analysis of the tracked items to transaction data produced from a transaction terminal to identify suspicious activity. Concretely, the step is performed by comparing a first frame and an empty base image, by subtraction and thresholding and the creation of a "binary object map", to identify new objects in the image that are not part of the empty base image, including an operator "object" and other objects within the region of interest.

40. The appellant stressed that it was not required to automate the process of D6 in order to identify transaction items. The claimed subject-matter extended beyond a mere automation of the process, since it required additional steps to arrive at the determination of the transaction items. There was no indication, in the prior art, that identification of objects could be obtained by subtraction of images followed by thresholding and the generation of a binary object map. Document D4 referred in general terms to techniques of thresholding in order to generate binary images. D4 was concerned with the identification of items based on characteristics that could be derived from video images. It did not suggest using these techniques to create object maps in order to identify objects entering or leaving a region of interest.

41. In the Board's judgement, the claimed subject-matter results in a straightforward manner from the initial decision of automating the process of D6. As stressed above, with regard to claim 1 of both the new main request and the first auxiliary request, no inventive step is involved in this initial step.

42. It would have been obvious for the skilled person, seeking to identify objects entering or leaving the region of interest, to rely on the dynamic inherent to the process, as it manifests itself in changes in the content between successive frames. Hence, it would have been obvious for the skilled person to use image processing techniques adapted for identifying such changes.

43. The Board has no doubt that the basic operations of subtracting images, and thresholding to create binary images were well-known in 2005. As stressed above, if this were not so, and in the absence of any details as to the implementation of these steps in application, an objection of insufficiency under Article 83 EPC would have been justified.

44. The issue of obviousness under Article 56 EPC appears thus to hinge on the question whether it would have been obvious, in the context of automating the process of D6, to use such known processes.

45. The comparison of frames by way of subtraction is straightforward in that it serves exactly the purpose of identifying changes between successive frames. It is particularly adapted to the present purpose of identifying objects entering, moving within, or leaving the region of interest. The comparison of an image with an empty base image (that is, an updated version of the background image (application, page 25, lines 18-23)) is a direct consequence of this finding. In order to allow reliable identification of transaction items, that is of objects entering, moving within, or leaving the region of interest, the skilled person would have recognised that processes of thresholding, to obtain a binary image of the region of interest, would have been particularly adapted to reducing the noise component in the resulting image.

46. The subject-matter of claim 1 according to the second auxiliary request is thus not inventive in the sense of Article 56 EPC.

Third auxiliary request - Inventive step (Article 56 EPC)

47. Similar considerations apply to the steps defined in claim 1 according to the third auxiliary request in order to obtain the video parameter.

48. The terminology is somewhat different, claim 1 defines a more general version of the comparison with a base image in claim 1 of the second auxiliary request. This is apparent in the following step:

defining a region of interest within the transaction area;

automatically identifying a first frame of video data that indicates a first set of items in the region of interest;

automatically identifying a second frame of video data that indicates a second set of items in the region of interest, the first set of items being visually distinct from the second set of items: and

automatically indicating the visual distinctness of the first set of items from the second set of items as an event indicating an item existed within the region of interest of the video data ....

In particular, the Board sees no difference in meaning between an "updated background image" (or "empty base image" in the terms of claim 1 according to the second auxiliary request) and a first frame of video data that indicates a first set of items in the region of interest.

49. The step of automatically indicating the visual distinctness of the first set of items from the second, as an event indicating that an item existed within the region of interest, refers to the actual result of the subtracting and thresholding processes envisaged above.

50. For the reasons developed above with regard to the second auxiliary request, the recited features would have been obvious.

51. The manual process of D6 also relies on the step of obtaining transaction data associated with the transaction terminal associated with the transaction area. The transaction data are indicative of items being registered as transaction items with the transaction terminal. The step of obtaining one transaction parameter means, in the context of the claimed invention, the automation of this step. It is thus a step already envisaged when deciding on automating the process of D6.

52. In D6, the step of identifying suspicious activity, when transaction data are missing for an item indicated as being within the region of interest, is performed by the operator. The method of the invention simply automates this step. Independently of those two facts, however, the Board stresses that identifying suspicious activity does no more than reflect the discrepancy between the two sets of data. As emphasised above, it is the result of a mere comparison of lists of events deprived of any technical character.

53. The subject-matter of claim 1 according to the third auxiliary request is thus not inventive in the sense of Article 56 EPC.

New fourth auxiliary request - Admissibility

54. The new fourth auxiliary request was filed after notification of the summons to oral proceedings, in reaction to the accompanying preliminary opinion. Thus, its admission is subject to Article 13(2) RPBA 2020 (cf. point 1 above) and the criteria of Article 13(1) RPBA 2020.

55. The new fourth auxiliary request attempts to address the Board's preliminary view that the subject-matter of claim 1 according to the fourth auxiliary request was not inventive.

56. The features added to claim 1 stem from original claims 5, 9, and 10. Claim 1, however, does not incorporate the features of original claims 15, 26, and 27 which were added in the new main request. While the newly added features indeed address the objection of lack of inventive step, claim 1 of the new fourth auxiliary request focuses on aspects different than those pursued in claim 1 of the new main request. The two sets of claims are not convergent.

57. The new fourth auxiliary request is thus not admitted into the appeal proceedings.

Fourth auxiliary request - Inventive step (Article 56 EPC)

58. Claim 1 of the fourth auxiliary request includes the features of identifying and transmitting to a reviewer for review a segment of the video data where the video event that is not identified as a transaction event exists.

59. The added steps contribute, in the appellant's opinion, to a reduction of data transmission or data storage. They define a technical feature of the claimed method.

60. The primary purpose of the added step is to allow a review of mismatches.

61. In the Board's judgment, the review suspicious activity identified in the course of the automated process of surveillance can be achieved in two alternative ways. Whether the segment of video data identified as referring to a possibly suspicious event is to be transmitted or consulted by the reviewer on the video terminal is merely a matter of preference. Neither of the alternatives is inventive.

62. Although transmission bandwidths were certainly more limited in 2005 than now, this would not constitute an obstacle for such transmission taking place. The Board is persuaded that the claimed alternative of identifying and transmitting a segment of the video data, would have been envisaged by the skilled person.

63. For this reason, the subject-matter of claim 1 according to the fourth auxiliary request is considered to be obvious in the sense of Article 56 EPC.

Auxiliary request 5 - Admissibility

64. Auxiliary request 5 was filed during oral proceedings before the Board, in reaction to the non-admission or non-allowance of all the previous requests. Thus, its admission is subject to Article 13(2) RPBA 2020 (cf. point 1 above) and the criteria of Article 13(1) RPBA 2020.

65. The arguments put forward by the appellant are not sufficient to decide favourably on the admission of auxiliary request 5. The circumstances referred to by the appellant are not exceptional.

66. The objections to lack of inventive step, set out in the Board's preliminary opinion, were along the same lines as in the Examining Division's refusal. Both the Examining Division (point 19 of the impugned decision) and the Board (points 16 and 34 of the preliminary opinion) referred to common general knowledge with regard to the general trend of automation and to certain aspects of image processing. It is not surprising that the debate during the oral proceedings concentrated on these aspects in more detail.

67. The Board has not provided direct evidence for the existence of the common general knowledge referred to. This is justified under the circumstances, in view of the content of the description, which refers to the image processing techniques in question without explaining them or showing any implementation. This is an unambiguous indication that they were well-known. As mentioned above, a different conclusion would have led to an objection under Article 83 EPC, since the skilled person would have been at a loss as to what the terms used to define the techniques actually implied, and would have been unable to implement them.

68. It follows that the arguments based on common general knowledge were to be expected. The appellant cannot rely on the element of surprise to justify the admissibility of auxiliary request 5.

69. The present circumstances are not exceptional, and do not justify the admission of auxiliary request 5 which is thus not admitted into the proceedings (Article 13(2) RPBA 2020).

70. In the absence of an allowable claim request, the appeal must be dismissed.

Order

For these reasons it is decided that:

The appeal is dismissed.