T 2703/18 (Object verification/UNIVERSITE DE GENEVE) of 6.12.2023

Summary of Facts and Submissions

I. The appeal is against the decision of the examining division, with reasons dispatched on 18 June 2018, to refuse Eur­­opean patent application No. 14160385.2 under Article 97(2) EPC and to not refund a further search fee paid under Rule 64(1) EPC.

II. The examining division refused the application on the basis that the main request and auxiliary requests 1 and 2 did not meet the requirement of an inventive step, Article 56 EPC, and because auxiliary request 3 was not admitted under Rule 137(3) EPC.

It rejected the request to refund a further search fee because it confirmed the search division's non-unity objection against the original set of claims.

III. The decision cited, inter alia, the following documents:

D1: M. Diephuis, "A Framework for Robust Forensic

Image Identification", Master Thesis, University

of Twente, 2010, XP055132956,

D2: WO 2012/126008 A2,

D3: US 2007/0286526 A1,

D4: R.J.M. den Hollander et al., "Logo recognition in

video by line profile classification", Storage

and Retrieval methods and Applications for

Multimedia 2004, SPIE Vol. 5307, 2004,

XP055132795,

D7: R. Szeliski, "Computer Vision: Algorithm and

Applications (September 3, 2010 draft)",

pages 685-691, XP055301588.

The examining division's reasoning referred only to D1, D2 and D7 (the latter not being identified in the decision itself but in the summons to oral proceedings).

IV. With the statement of grounds of appeal, the appellant requested that the decision of the examining division to refuse the application be set aside and that a patent be granted on the basis of the main request or, alternatively, one of auxiliary requests 1 to 4 filed with the statement of grounds of appeal.

The appellant also requested a refund of the further search fee paid for original claims 10 and 11 and a reimbursement of the appeal fee due to a violation of the appellant's right to be heard, Article 113(1) EPC.

V. In the annex to the summons to oral proceedings, the board presented its preliminary opinion on the appeal. The following prior art document was introduced in accordance with Article 114(1) EPC:

D8: S. Voloshynovskiy, M. Diephuis et al., "On

accuracy, robustness, and security of bag-of-

word search systems", Proc. SPIE 9028, 902807,

19 February 2014, XP093091740.

The first two authors of D8 are the inventors of the present application. D8 is reference [137] in the later-published PhD thesis of the second author, to which the appellant drew the attention of the board in the grounds of appeal (page 26, fifth complete paragraph).

Several objections under Articles 84 and 123(2) EPC were raised by the board. The board noted that it was not convinced by the examining division's inventive step lines of argument starting from either D1 or D2. However, claim 1 of the main request and auxiliary requests 1 and 2 did not appear to involve an inventive step starting from D2 combined with common general knowledge as illustrated by D8. Auxiliary requests 3 and 4 appeared not to be allowable for lack of compliance with Article 123(2) EPC. The board tended to consider that the refund of the further search fee paid was to be granted but not the reimbursement of the appeal fee.

VI. With a letter dated 3 December 2023, the appellant filed amended claims of a new main request and new auxiliary requests 1 to 3, and an amended description.

VII. At the oral proceedings, the appellant filed a new set of claims 1 to 15 and amended description pages 1/42 to 42/42 according to a new main request. All previous requests were withdrawn except the requests that one further search fee be refunded and that the appeal fee be reimbursed. At the end of the oral proceedings, the chairman announced the decision of the board.

VIII. Claim 1 as originally filed reads as follows:

Method for recognition and/or verification of authenticity of digital - and/or physical objects comprising the steps of codebook training, object enrollment, and object verification, the latter consisting in object recognition and/or object authentication,

- the codebook training step comprising

o providing (210) a training set of images

x(1),...,x(T) consisting of T training images,

o extracting (2, 6) for each image x(1),...,x(T) at

least one set of training features

(xi(1),...,xi(T),xa(1),...,xa(T)),

o obtaining at least one set of centroids for said

at least one set of training features

(xi(1),...,xi(T),xa(1),...,xa(T)),

o registering (3, 7) said at least one set of

centroids in at least one codebook (Ci, Ca),

- the object enrollment step comprising

o acquiring (1) an image x(m) of an object x to be

protected,

o extracting (2, 6) at least one set of features

(xi(m), xa(m)) from image x(m),

o enrolling (4, 8) and storing (5, 9) said at least

one set of features (xi(m), xa(m)) in at least

one database (Di, Da),

o repeating said acquiring (1), extracting (2, 6),

and enrolling steps (4, 5, 8, 9) for each object

to be protected,

characterised by the fact that

- the object verification step comprises

o acquiring (10) an image y of an object under

verification,

o extracting (2, 6) at least one set of features

(yi, ya) from image y,

o identifying (11) object y by producing an

estimate of the object index m' referring to the

at least one database (Di) comprising at least

one of said sets of enrolled features (xi(m)),

o authenticating (12) the object by comparing at

least one set of features (y'a(m')) of the

acquired image y with at least one set of

enrolled features (xa(m')) obtained from at least

one of said databases (Da),

o repeating said extracting (2, 6), identifying

(11), and authenticating steps (12) for each

object under to be recognized and/or

authenticated,

and by the fact that the method uses two types of features (xi(m), xa(m)), of codebooks (Ci, Ca), and of databases (Di, Da) which are specifically designed for identification, respectively authentication, the identification database (Di) and authentication database (Da) having different organizations.

Claim 3 as originally filed reads as follows:

Method according to one of the preceding claims, characterised by the fact that

- the codebook training step comprises

o providing (210) a training set of images

x(1),...,x(T) consisting of T training images,

o extracting (2, 6) for each image x(1),...,x(T)

identification features (xi(1),...,xi(T)) and

authentication features (xa(1),...,xa(T)),

o applying (212, 213, 214, 215) vector quantizers

to said identification features (xi(1),...,xi(T))

and authentication features (xa(1),...,xa(T)) to

obtain corresponding sets of identification and

authentication centroids,

o registering (3, 7) said sets of centroids in an

identification codebook (Ci) and an

authentication codebook (Ca),

- the object enrollment step comprising

o acquiring (1) an image x(m) of a digital - or

physical object x to be protected with the help

of an acquisition device, in particular with the

help of a mobile phone equipped with a camera,

directly from the physical object or from an

electronic file,

o extracting (2, 6) identification features xi(m)

and authentication features xa(m) from image

x(m),

o enrolling (4, 8) and storing (5, 9)

identification features xi(m) in an

identification database (Di) and authentication

features xa(m) in an authentication database

(Da),

o repeating said acquiring (1), extracting

(2, 6), and enrolling steps (4, 5, 8, 9) for each

object to be protected,

- the object verification step comprising

o acquiring (10) an image y of an object under

verification with the help of said acquisition

device,

o extracting (2, 6) identification features (yi)

and authentication features (ya) from image y,

o identifying (11) object y by producing an

estimate of the object index m' referring to the

identification database (Di) of enrolled

identification features (xi(m')) by matching the

identification features (yi) extracted from the

acquired image y with the identification features

(xi(m')) stored in the identification database

(Di) using a fast matching procedure via an

inverted file,

o aligning (402) the authentication features

(ya(m')) extracted from the acquired image y with

the enrolled authentication features (xa(m')) of

the object estimate m', resulting in aligned

authentication features (y'a(m')),

o authenticating (12) the object by comparing the

aligned authentication features (y'a(m')) of the

acquired image y with the enrolled authentication

features (xa(m')) obtained from the

authentication database (Da) with the help of the

estimate of the object index m',

o repeating said extracting (2, 6), identifying

(11), aligning (402), and authenticating steps

(12) for each object to be recognized and/or

authenticated.

Claim 10 as originally filed reads as follows:

Method according to one of the preceding claims 3 to 9, characterised by the fact that the identification features xi(m) extracted from image x(m) during the enrollment step and the identification features (yi) extracted from an acquired image y are encoded by applying multiple assignment by creating overlapping regions (2321, 2322, 2323) at the separation boundaries between neighboring centroids (c**(i), c**(l), c**(n)) and attributing features (x**(k)) belonging to these overlapping regions (2321, 2322, 2323) to all of the corresponding centroids (c**(i), c**(l), c**(n)) both in the encoding (2330) and decoding steps (2350), the overlapping regions being defined with the help of a reliability function R**(in(k))= Q(((x**(k))**(T)(c**(n)- c**(i)) - d**(in)/2)/sigma) corresponding to the probability of flipping a given feature X**(k)from centroid c**(i) to centroid c**(n), where (x**(k))**(T)(c**(n)- c**(i)) represents a projection of x**(k)onto the line (c**(n) - c**(i)) and d**(in)/2 corresponds to the half distance between two centroids with the Euclidian distance between two centroids c**(n)and c**(i) being defined as d**(in)=?c**(n)- c**(i)?, said reliability function R**(in(k))being used as a weight at least for the features near the boundaries between the centroids of neighboring centroids (c**(i), c**(l), c**(n)).

Claim 11 as originally filed reads as follows:

Method according to one of the preceding claims 3 to 10, characterised by the fact that aligning (402) the authentication features (ya) extracted from the acquired image y with the enrolled authentication features (xa(m')) of the object estimate m', resulting in aligned authentication features (y'a), is performed by partitioning the acquired image y and aligning each partition with a corresponding part of a template image corresponding to the object estimate m' by use of a local projective transform based on local features of each partition.

IX. Claim 1 of the (final) main request reads as follows (itemisation in bold by the board):

1. Method for verification of authenticity of

physical objects

1.1 comprising the steps of codebook training, object

enrollment, and object verification, the latter

consisting in object identification and object

authentication,

2. - the codebook training step comprising

2.1 o providing (210) a training set of images

x(1),...,x(T) consisting of T training

images,

2.2 o extracting (2, 6) for each image

x(1),...,x(T) identification features

(xi(1),...,xi(T)) and authentication

features (xa(1),...,xa(T)),

2.3 [o] obtaining corresponding sets of

identification and authentication centroids

for said identification features

(xi(1),...,xi(T)) and authentication

features (xa(1),...,xa(T)),

2.4 o registering (3, 7) said sets of centroids

in an identification codebook (Ci) and an

authentication codebook (Ca),

3. - the object enrollment step comprising

3.1 o acquiring (1) an image x(m) of an object x

to be protected directly from said object

by use of an acquisition device or from an

electronic file,

3.2 o extracting (2, 6) identification features

xi(m) and authentication features xa(m)

from image x(m),

3.3 o enrolling (4, 8) and storing (5, 9) said

identification features xi(m), by use of

the previously trained identification

codebook (Ci), in an identification

database (Di) as enrolled identification

features and said authentication features

xa(m), by use of the previously trained

authentication codebook (Ca), in an

authentication database (Da) as enrolled

authentication features,

3.4 o repeating said acquiring (1), extracting

(2, 6), and enrolling steps (4, 5, 8, 9)

for each object to be protected,

4.1 wherein the method uses two sets of features

(xi(m), xa(m)) and two codebooks (Ci, Ca) for

identification, respectively authentication, the

identification features and codebook and the

authentication features and codebook

corresponding to different levels of granularity,

4.2 and two types of databases (Di, Da) which are

specifically designed for identification,

respectively authentication, the identification

database (Di) and authentication database (Da)

having different organizations,

4.3 wherein the identification database (Di) is

designed as an inverted file containing for each

enrolled identification feature (xi(m)) the

corresponding indices (m) of objects possessing

this feature, whilst the authentication database

(Da) is designed as a lookup table storing at

index m the enrolled authentication features

(xa(m)) of an object with an attributed index m,

5. wherein - [sic] the object verification step

comprises

5.1 [-] acquiring (10) an image y of an object under

verification by use of a mobile acquisition

device, in particular by use of a mobile phone

equipped with a camera,

5.2 - extracting (2, 6) identification features (yi)

and authentication features (ya) from image y,

5.3 - identifying (11) object y by producing the

object index m' of an enrolled object that most

likely corresponds to the object under

verification by matching the identification

features (yi) extracted from the acquired image y

with the identification centroids registered in

the identification codebook (Ci) and forwarding

the matched identification centroids to the

identification database (Di) to identify the

object index m', said object index m' being

forwarded to the authentication database (Da),

5.4 - aligning (402) the authentication features

(ya(m')) extracted from the acquired image y with

the enrolled authentication features (xa(m')) of

the enrolled object with object index m' obtained

from the authentication database (Da), resulting

in aligned authentication features (y'a(m')),

5.5 - authenticating (12) the object under

verification by comparing the aligned

authentication features (y'a(m')) extracted from

the acquired image y with the enrolled

authentication features (xa(m')) of the enrolled

object with object index m' obtained from the

authentication database (Da),

5.6 - repeating said acquiring (10), extracting (2,

6), identifying (11), aligning (402), and

authenticating steps (12) for each object under

verification,

wherein

6.1 extracting the identification features xi(m) and

authentication features xa(m) from images x(m)

acquired from enrolled objects as well as

extracting the identification features (yi) and

authentication features (ya) acquired from images

y from objects under verification comprises use

of key points and descriptors,

6.2 each descriptor corresponding to an individual

key point or to a group of key points,

6.3 wherein at least the identification features

xi(m) extracted from images x(m) acquired from

enrolled objects and the identification features

(yi) extracted from images y acquired from

objects under verification provide a non-local

image characterization by comprising multiple key

points descriptors providing a descriptionof

[sic] features extracted along a connecting line

between key points satisfying pre-defined

constraints on geometrical parameters of key

point locations and their descriptors.

In the following, the parts of a feature concerning only identification or authentication will be referred to by adding "(i)" or "(a)" to the feature label, e.g. "feature 5.2(i)" would refer to the step of extracting identification features from image y, and "feature 5.2(a)" to the same step for authentication features, both features 5.2(i) and 5.2(a) being sub-features of feature 5.2.

Claim 13 of the main request reads as follows:

Computer program means stored in a computer readable medium adapted to implement the method according to one of the preceding claims.

Claim 14 of the main request reads as follows:

Device equipped with computer program means according to the preceding claim, wherein the device is chosen from the group comprising a mobile phone, in particular a smart phone equipped with a camera, a digital photo apparatus, a digital video camera, a barcode reader equipped with a camera, a scanning device.

Reasons for the Decision

The application

1. The application relates to the protection of physical objects (e.g. packaging) against counterfeiting by automatically verifying their authenticity (page 1, lines 13-24, of the original description).

2. The "object verification" phase, as carried out in the main embodiment, is illustrated in figure 4:

FORMULA/TABLE/GRAPHIC

An image y of an object under verification is acquired (step 10, "probe image y"), e.g. by a customer using the camera of a mobile phone (page 17, lines 19-23).

In an "identification" stage (2, 11), one or more candidate enrolled objects are determined on the basis of "identification features" extracted from image y and identification features of enrolled objects previously stored in quantized form in an "identification database" (5) using an "identification codebook" (3) (page 17, line 23 to page 18, line 14; page 18, lines 20-22; page 20, lines 10-17; figure 5).

In an "authentication" stage (6, 402, 12), it is then verified whether image y actually corresponds to any of the determined candidate enrolled objects. This is performed on the basis of "authentication features" extracted from image y and authentication features of the enrolled objects previously stored in quantized form in an "authentication database" (9) using an "authentication codebook" (7) (page 18, line 14 to page 19, line 4). This stage involves a step of aligning (402) the authentication features of the probe image with those of a candidate enrolled object (page 33, lines 1-27).

The probe object is considered authentic only if, in the identification stage, a sufficiently similar enrolled object is found that passes the comparison test vis-à-vis the probe object in the authentication stage (page 20, lines 12-17; page 35, lines 16-28).

3. In the main embodiment, the features, codebooks and databases are designed so that the identification stage amounts to a coarse but fast matching of image y with all enrolled objects in the database and the authentication stage to a fine comparison of image y with only a few candidate enrolled objects (page 14, lines 17-19; paragraph bridging pages 15 and 16; page 18, lines 12-14; page 34, lines 25-27).

4. To enable a fast matching during the identification stage, the identification database is organised as an inverted file: for given indices of codewords (learned in a preliminary codebook training phase as centroids of identification features of training images), it re­turns the indices of enrolled objects having identi­fi­cation features that map to these codewords (page 17, lines 10-13; page 19, line 25 to page 20, line 4).

The authentication database is organised as a lookup-table that, for a given index of an enrolled object, returns the (quantized) authentication features of that object (page 17, lines 13-14).

5. Suitable features for identification and authentication are described, for instance, from page 20, line 19 to page 23, line 3 and in figure 6. Their extraction involves an optional global alignment (16) followed by an extraction of key points (17) and a computation of descriptors (18).

6. The authentication features may be extracted from the images in the same manner as the identification features, except at a lower level of granularity (page 34, lines 19-27), e.g. by partitioning the image into blocks and extracting features from each block individually (page 35, lines 1-14).

7. The alignment step (402) in the authentication stage may be based on block-wise local projective appro­xi­mations, which is said to account for distortions typical of lenses used in mobile phones (page 37, lines 22-27).

8. In a preferred embodiment, features for identification and/or authentication are extracted as illustrated in figure 19 (see page 27, line 23 to page 30, line 24):

FORMULA/TABLE/GRAPHIC

First, local descriptors (such as those shown in fi­gure 11) are extracted. A pair of keypoints (1501, 1502) satisfying pre-defined constraints on the geometric parameters of keypoints and their descriptors is identified. A cross-section (1505) of the image is extracted along the line connecting the two keypoints. A histogram representing the cross-section is then transformed (1506, 1507, 1508) into a signal (1509) which is quantized (1510) to produce a binary index (with respect to the quantizer's codebook 1512).

The board understands that in this case the signal (1509) represents an "authentication feature" (or an "identification feature") and the binary index represents the quantized version of that feature that is actually stored when it is said that the authentication feature is stored "using the authentication codebook" (page 35, lines 1-14).

9. This particular type of identification and authen­tica­tion features and their encoding are said to be more discriminative than local descriptors and particularly robust against rotation and the kind of projective transformations that may arise when the probe image is taken with the camera of a mobile phone (page 24, lines 24-26; page 28, lines 12-15 and 20-21; page 29, lines 15-24; page 30, lines 11-24). This avoids the need for an accurate alignment at the identification stage (page 20, line 24 to page 21, line 11; page 33, lines 2-6).

10. The proposed method is said to be "able to clearly detect any non-correspondence in images on microscopic level even when the images are acquired in non-syn­chro­nized way by mobile phone" (page 39, lines 4-7).

11. Compared to the prior art acknowledged on pages 1-7, the proposed method is said to enable inter alia the protection of "generic physical or digital objects" and to be "fast, robust, accurate and secure", without requiring a modification of the objects to be protected (page 7, line 22 to page 8, line 8; page 41, line 29 to page 42, line 10).

The claims of the main request and the amended description where it refers to the invention no longer refer to digital objects and are limited to the protection of physical objects (see e.g. claim 1 of the main request, line 1, and page 1, first paragraph, and the paragraph bridging pages 7 and 8 of the amended description).

Admittance

12. The application documents on which the main request is based were filed during the oral proceedings before the board. They overcome all the objections under Articles 84 and 123(2) EPC that had been raised for the first time by the board in its preliminary opinion and, more generally, comply with the requirements of the EPC. The board decided therefore to exercise its dis­cretion under Article 13(2) RPBA to admit this request.

13. As regards compliance with the requirements of Article 123(2) EPC, the board notes that claim 1 of the main request is based on original claims 1 to 6 with clarification of several features based on the corres­ponding passages of the original description cited above.

Inventive step

14. In the contested decision, the examining division relied on D1 and D2 as alternative starting points for objections of lack of inventive step against earlier (and broader) versions of claim 1. In the following, it is shown that claim 1 involves an inventive step over both D1 and D2.

15. Document D1

15.1 D1 is the master thesis of one of the inventors of the present application. D1 discloses a "framework" for using surface microstructures of physical objects, e.g. a packaging, for determining whether the object is authentic or counterfeited. It is based on the observation that surface microstructure is unique, much like a human fingerprint, and unclonable (section 1.1).

15.2 The proposed approach is illustrated in figure 1.1:

FORMULA/TABLE/GRAPHIC

It is assumed in D1 that all considered objects (the enrolled ones as well as those to be verified) display, on their surfaces, a shape of interest, e.g. a parti­cular logo. All images are taken such that the shape is in the field of view (section 2.1, second paragraph; section 2.2, second paragraph). The region of the object surface whose microstructure will be analysed ("region of interest") is defined with respect to that shape (section 1.2.2).

A "template", which appears to be a reference image of that shape under optimal imaging conditions, is avai­lable. Given an image of an object to be verified, the template is used to locate the shape, taking account of possible distortions ("synchronisation"), and to iden­tify the region of the image from which the micro­struc­ture is to be extracted ("microstructure extraction") (section 2.1, first and second paragraphs).

15.3 In an "enrollment" stage, an image of an object to be enrolled is acquired in the factory with an industrial camera such that the shape is in the center of the image. No synchronisation with the template is thus nee­ded. A patch of the image corresponding to the relevant region is extracted ("extract microstructure"). Features are extracted from the patch ("binarized microstructure") and stored in a database (section 1.2.1, 1.2.2).

In a "verification" stage, an image of an object to be verified is acquired with a portable device, e.g. a portable microscope. The shape may be anywhere in the image, possibly distorted. The template is used to perform an "image synchronisation", i.e. a transforma­tion of the image to remove the distortion, and to identify the region from which the microstructure patch is to be extracted. Features are then extracted from that patch. The resulting "binarized microstructure" is used as query for a nearest-neighbor search in the data­base (this search is referred to as "identifica­tion" in D1, section 1.2.5). If there is a match with the binarized microstructure of an enrolled object, the object under verification is considered authentic.

15.4 The proposed synchronisation algorithm is illustrated in figures 2.8 and 2.9.

SIFT features are chosen for their invariance to lighting and acquisition conditions (section 2.3.5). Their use involves the detection of SIFT keypoints and the computation of SIFT descriptors (vectors) for each keypoint (figure 2.9(b); section 2.4, introduction; section 2.4.1 and 2.4.2).

The SIFT descriptors extracted from the object under investigation are matched to the SIFT descriptors of the template in an exhaustive nearest neighbor search (figure 2.9(c); section 2.4.2, second and third paragraphs). Erroneous matches are discarded by Hough pose-space clustering and by a RANSAC algorithm (figure 2.9(d); section 2.4, introduction; section 2.4.3, last paragraph).

Given the remaining robust matches, the Discrete Linear Transform is used to generate a projective trans­for­mation to synchronise the image with the template (section 2.4.3; annex B.6).

15.5 As regards the extraction of features from micro­structure patches, several options are discussed in chapter 5.

15.5.1 It is explained that a basic approach is to compare microstructure patches by cross-correlation, hence without previous feature extraction, but that this is time-intensive. Therefore, D1 considers features to be extracted from the microstructure patches that achieve a dimensionality reduction but retain sufficient information for the verification task (section 5.1).

15.5.2 The preferred approach is the extraction of "reliable components" from the microstructure patches using ran­dom projections and magnitude sorting, as explained in particular in sections 5.4 and 5.4.1, with reference to figure 5.7.

Each microstructure patch (for enrolled or probe objects) is represented by two vectors of same fixed length n (e.g. n = 32): a "reliable bit position" vector and a "reliable components" vector. During the enrollment stage, the bit positions vector and reliable components vector of each enrolled object are separa­tely stored (see figure 5.7(a): "Reduced codebook" and "Bit position" databases). For a probe object, the bit position vector of the microstructure patch is used to identify those enrolled objects having the same bit position vectors as "candidates". The reliable compo­nents vector of the probe object is then only compared to the reliable components vectors of the candidate enrolled objects (see figure 5.7(b)).

This approach is said to "drastically limit the search space in which the identification is being resolved" (section 6.3, last paragraph).

15.6 D1 also makes the general remark that "metadata" and "smart coding techniques" involving an index file kept in memory may be used to speed up the identification process (section 1.2.5, third paragraph).

16. Comparison of claim 1 with D1

16.1 D1 discloses thereby, in the terms of claim 1, a method for verification of authenticity of physical objects (feature 1) comprising steps of object enrollment and object verification (part of feature 1.1).

16.2 The examining division's reasoning relied on mapping the SIFT features (keypoints and descriptor vectors) extracted in D1 to the identification features of claim 1, and the "microstructure patches" extracted in D1 to the authentication features of claim 1 (decision, points 1.1 and 1.3.2, and point 1.4, explanations for features 2.2 and 3.2).

The examining division noted that in D1 the category of the object to be verified was assumed to be known "before authentication", as only one template was used, but that it would have been obvious to consider "the problem of automating the task of knowing the object before authentication, i.e. the task of object recog­nition". It would furthermore have been obvious that the SIFT features used in D1 for alignment could also be used for that recognition task. The "only required adap­tation [would be] correspondingly training the code­book". D7 was cited in support of these assertions (decision, point 1.5.1).

16.3 The board is not convinced by the examining division's feature mapping and the associated obviousness reasoning.

16.3.1 The SIFT features are used in D1 for alignment, not for identification. They are only extracted for probe objects, not for enrolled objects. The SIFT features of a probe object are matched with SIFT features of the template, not with SIFT features of enrolled objects stored in an (identification) database.

Hence, if this feature mapping were adopted, at least feature 5.3 in combination with features 3.2(i) to 3.4(i) would be missing. It is noted that these features were already present in the version of claim 1 considered in the examining division's decision (in a slightly different wording).

16.3.2 As regards obviousness, the board fails to see that the suggested obvious modification of the method of D1 would have resulted in a method including all claim features in combination, in particular feature 5.3 in combination with features 3.2(i) to 3.4(i).

16.3.3 The examining division did not elaborate on the con­crete modifications to be carried out nor on how the added steps would connect to the other steps of the method of D1.

16.3.4 In the board's view, it appears that an obvious approach to solve the aforementioned problem starting from D1 would have been to store templates (i.e. their SIFT features) for different object categories and, for each template, to enroll objects comprising that template in the database. Given a probe image, a first step would be to determine the appropriate template. Once determined, the method could continue as in D1, with the search into the database being limited to those enrolled objects corresponding to the determined template. It appears also that it would have been obvious to consider automating the determination of the appropriate template by using a SIFT matching approach, as suggested by the examining division, i.e. by separa­tely attempting to align the probe image with each template as in the synchronisation step of D1 and to keep the template for which this alignment is most successful.

In this approach, the step of determining the appropriate template would involve matching the SIFT features of the probe image with the SIFT features of templates - not SIFT features of enrolled objects, as would be required to meet feature 5.3 in combination with features 3.2(i) to 3.4(i).

16.3.5 It is noted that if the examining division meant the initial object recognition step to be carried out in­dependently of templates, as a general object cate­gory recognition step preceding the synchronisation step with a template appropriate for the determined cate­gory, it is not apparent why such a recognition step would involve a matching with SIFT features of enrolled objects and why it would result in an object index m' (rather than a template index identifying the template to be used for subsequent object identification and authentication), again as required by feature 5.3 in combination with features 3.2(i) to 3.4(i).

16.3.6 The examining division's sweeping reference to D7 (decision, point 1.5.1: "it is commonly known that the SIFT method used for alignment in D1 is in general used to recognize and align objects - see e.g. D7 p.690 or p.685, 686 sect. 14.3 and par.3, sect. 14.3.1 par.3; the only required adaptation is corresponding training the codebook - see again D7, passages previously indi­ca­ted") does not address the specific issues raised above.

16.3.7 Already for these reasons, the examining division's feature mapping and obviousness reasoning were not convincing for the version of claim 1 underlying the decision (see the board's communication dated 23 Octo­ber 2023, point 34.2.5, for further considera­tions), and a fortiori they are not convincing for present claim 1, which includes further features not disclosed in D1 (e.g. features 5.4 and 6.3).

16.4 The board considers in the following an alternative feature mapping between the disclosure of D1 and claim 1.

16.5 The "identification" stage in D1 may be mapped to the "object verification" stage of claim 1. The embodiment of D1 in which the identification stage is implemented with random projections and reliable components (see D1, section 5.4.1 and figure 5.7; see also point 15.5.2 above) involves two sub-stages that can be mapped to the "object identification" and "object authentication" stages of claim 1.

16.6 The use of bit positions vectors and reliable components vectors in this embodiment serves to perform a fast matching based on a coarse/fine strategy (see D1, page 104, sub-section "Fast searching", and figure 5.7(b): "using reliable query bits to reduce the number of candidates prior to decoding"), hence essentially the same purpose as the identification/authentication stages in the method of claim 1.

The bit positions vectors may be mapped to the "iden­ti­fication features" and the reliable components vectors to the "authentication features" of claim 1, the "bit positions" database to the "identification database" and the "codebook" to the "authentication database".

16.7 Based on this feature mapping, the method of claim 1 differs from that disclosed in D1 at least in the following features:

(i) the identification and authentication features

use key points and descriptors, as specified in

features 6.1 and 6.2;

(ii) at least the identification features provide a

non-local image characterization by comprising

multiple key points descriptors providing a

description of features extracted along a

connecting line between key points satisfying

pre-defined constraints on geometrical

parameters of key point locations and their

descriptors, as specified in feature 6.3;

(iii) identification and authentication codebooks are

generated as specified in features 2 to 2.4

and used as specified in features 3.3, 4.1 and

5.3;

(iv) the identification and authentication databases

have different organizations, as specified in

features 4.2 and 4.3, which are used as

specified in features 5.3 to 5.5;

(v) the authentication features extracted from the

acquired image y are aligned with the enrolled

authentication features of the enrolled object

with object index m', as specified in

feature 5.4, before they are being compared to

them in the authentication step (feature 5.5).

16.8 In view of D1, the differentiating features may be considered to solve the technical problem of providing an alternative, computationally efficient method for verifying the authenticity of a physical object that does not require the use of a template.

16.9 D1 does not provide any hint as to how templates could be dispensed with.

16.10 Distinguishing features (i) and (ii) define a different type of image features to be extracted at least for identification.

The board notes that D3 and D4 were cited in the European search opinion for original claim 6, which comprised claim features similar to (i) and (ii). D3 (paragraphs [0040]-[0044] and [0054]-[0064]; figures 2A to 2E) discloses a multi-point descriptor called "2-point EDGE" based on the pixel intensity profile along a line connecting two points of interest. It is used to match geometrical patterns in two images (see e.g. figure 1). D4 (sections 1 and 3; figure 2) disclo­ses detecting a logo in an image based on line intensi­ty profiles.

It is not apparent why the skilled person would have replaced the bit positions and reliable components vectors used in D1 by such multi-point descriptors. These vectors are used in D1 for matching extracted microstructures, which are not assumed to exhibit geometrical patterns (see D1, page 3, figure 1.1). D3 and D4 do not disclose using pixel intensity lines for matching microstructures. In D1, the detection of a geometrical pattern like a logo is part of the "synchro­nisation" that precedes the extraction of a microstructure and involves only the probe image and the template, not images of enrolled objects (see again figure 1.1).

Furthermore, even if the skilled person had considered using pixel intensity lines as disclosed in D3 and D4 to match extracted microstructures in the method of D1, this would have resulted in replacing the computation of bit positions and reliable components vectors by the computation of pixel intensity lines, and the resulting method would no longer have involved distinct identification and authentication stages, as required by claim 1.

The skilled person starting from D1 would thus not have arrived at distinguishing features (i) and (ii) in combination with the other features of claim 1.

16.11 Additionally, the board notes that the components of the bit positions vectors are ordinal numbers defining an ordering (see D1, page 106, figure 5.7(a)). Centroids of such vectors are no longer vectors defining an ordering. Hence, there would be no reason for the skilled person starting from D1 to generate and use an "identification codebook" as specified in features 2, 2.1(i) to 2.4(i), 3.3(i), 4.1(i) and 5.3 for the bit positions vectors in D1.

The skilled person starting from D1 would thus not have arrived at distinguishing feature (iii) in combination with the other features of claim 1.

16.12 Furthermore, as the reliable components vectors used in D1 no longer have a direct relationship with the geome­try of the patch, it is not apparent how they could be "aligned" as required by differentiating feature (iv). In the method of D1, an alignment is carried out beforehand, via the synchronisation with a template.

The skilled person starting from D1 would thus not have arrived at distinguishing feature (v) in combination with the other features of claim 1.

17. Document D2

17.1 D2 discloses a method for the verification of physical objects based on their (unique) microscopic texture (paragraphs [0002], [0007]).

17.2 "Microscopic texture images" (hereinafter: "texture images") of objects may be acquired by a microscope attached either to an articulated arm (paragraph [0054]) or to a mobile phone (paragraph [0057]). When the images of two objects are compared, it is assumed that the textures have the same orientation in both images (see paragraph [0060]). That this is the case is ensured during acquisition of the texture images on the basis of a marker (e.g. a logo) present on the surface of the objects (paragraphs [0053], [0057], [0058], [0060], [0063]).

17.3 Each texture image is represented by a set of extracted local descriptors (features), each comprising a keypoint and an associated 128-dimensional descriptor vector, which is a histogram of gradient orientations around the neighborhood of the keypoint (paragraph [0060]). The extracted local descriptors represent a "fingerprint" for the object (paragraph [0007]).

17.4 To determine whether two microscopic textures correspond to the same surface, the local descriptors extracted from their texture images are matched. First, this involves exhaustively matching the descriptor vectors of both texture images based on their Euclidean distances. For each pair of matching descriptors, their "slope differences" are computed (this takes the keypoint locations into account). If 80% of the slope differences is within a given threshold, the two texture images are considered to match; otherwise not (paragraphs [0062], [0063], figures 3(a) and 3(b)).

17.5 In the context of quantifying the entropy of a microscopic texture, it is suggested to quantize the 128-dimensional descriptor vectors so as to ensure that multiple readings of the same microscopic texture, which may slightly vary, result in the same quantized descriptor vector (paragraphs [0074], [0076], [0077], [0079]). The quantization is carried out by a scalar normalization followed by rounding the value to the nearest integer (paragraph [0078], last sentence). It appears that the purpose of these entropy calculations in D2 is to establish the general suitability of microscopic textures as fingerprints for physical surfaces (paragraph [0083], last sentence).

17.6 Several application fields are mentioned, e.g. authentication of artworks, bank checks, clothes.

It is suggested to print on a physical object a compact encoding of its fingerprint (e.g. as a barcode). A user may then authenticate the physical object offline with a mobile phone-microscope assembly, by acquiring a tex­ture image of the object, computing its local descrip­tor as a fingerprint, reading and decoding the barcode to obtain a reference fingerprint, and checking whether the two fingerprints match (paragraphs [0084]-[0088]).

Alternatively, the reference fingerprint may be re­trieved from an online repository storing fingerprints for verifying the authenticity of various physical objects (paragraph [0086], last sentence; paragraph [0134], [0135]; claim 25). D2 does not indicate how the reference fingerprint is to be determined in that case.

18. Comparison of claim 1 with D2

18.1 The examining division's line of argument starting from D2 (decision, point 2) is very brief, essentially indicating that it is "similar to that starting from D1" (decision, point 2.1).

18.2 Taking as starting point the embodiment of D2 in which the reference fingerprints are stored in an online repository, the board considers that D2 discloses a method for verification of the authenticity of physical objects comprising steps of object enrollment (storage of reference fingerprints in the online repository) and object verification (comparison of local descriptors extracted for a given object with a reference fingerprint). The microscopic texture image acquired from an object with a mobile-phone-microscope assembly may be mapped to an "image of an object" in the sense of claim 1, the local descriptors (each comprising a key point and an associated 128-dimensional descriptor vector) extracted from the microscopic texture image to "authentication features" of that object, the reference fingerprints stored in the online repository to "enrolled authentication features" stored in an "authentication database".

18.3 Based on this feature mapping, the method of claim 1 differs from that disclosed in D2 at least in the following features:

(i) the enrolled authentication features with which the extracted authentication features are compared are determined on the basis of the result of a step of identifying object y by producing the object index m' of an enrolled object that most likely corresponds to the object under verification by matching identifi­ca­tion features extracted from the acquired image with enrolled identification features of enrolled objects stored in an identification database using an identi­fi­cation codebook as specified in features 5.2(i) and 5.3 in combination with features 3.2(i) to 3.4(i);

(ii) the authentication features extracted from the acquired image y are aligned with the enrolled authentication features of the enrolled object with object index m', as specified in feature 5.4, before they are being compared to them in the authentication step (feature 5.5).

(iii) the generation of identification and authenti­ca­tion codebooks as specified in features 2 to 2.4 and their use in the object enrollment and object verifica­tion steps as specified in features 5.3 and 5.5;

(iv) the identification and authentication features and the identification and authentication codebooks corres­pond to different levels of granularity as specified in feature 4.1;

(v) the identification and authentication databases have different organizations, the identification database being designed as an inverted file and the authentication database as a lookup table as specified in features 4.2 and 4.3, which are used as specified in features 5.3 and 5.5;

(vi) at least the identification features provide a non-local image characterization by comprising multiple key points descriptors providing a description of fea­tures extracted along a connecting line between key points satisfying pre-defined constraints on geome­tri­cal parameters of key point locations and their descriptors, as specified in feature 6.3.

18.4 The differentiating features may be considered to solve over D2 the technical problem of providing an alter­native, computationally efficient method for verifying the authenticity of a physical object.

18.5 D2 does not disclose how to proceed when the reference fingerprints are stored in an online repository. In its preliminary opinion, the board considered that it might have been obvious for the skilled person starting from D2 to identify the most relevant reference fingerprint stored in the online repository with which the local descriptors of the object under verification would then have to be compared by adopting a coarse-fine approach using the bag-of-words approach and vector quantization as described in D8 (as they were suitable for the SIFT-like features used in D2) and by organising the resulting identification and authentication databases as an inverted file and a lookup table, respectively (as also described in D8, figure 2). This would have essentially amounted to distinguishing features (i) and (iii) to (v).

18.6 Whether that would indeed have been obvious may be left open, as the board considers that it would in any case not have been obvious for the skilled person, starting from D2, to arrive at the additional distinguishing features (ii) and (vi) of present claim 1 over D2.

18.7 An alignment of the local descriptors prior to comparison (feature (ii)) would appear to be pointless starting from D2 as D2 assumes the images to have been acquired so that the textures have the same orientation in both images. That this is the case is ensured manu­ally during acquisition of the texture images on the basis of a marker (e.g. a logo) assumed to be present on the surface of the objects (see D2, paragraph [0060]).

It is also noted that the obvious approach to automate this aspect of the acquisition of the images would have been to include an automatic alignment step prior to object identification and authentication, essentially as done in D1 on the basis of a template ("synchroni­sa­tion"). This would also not have lead to distinguishing feature (ii) in combination with the other claim features.

18.8 As regards distinguishing feature (vi), the board notes, as in point 16.10 above in respect of D1 as starting point, that D3 and D4 do not disclose using pixel intensity lines for matching microstructure (texture) images, as required in D2. Hence, the skilled person starting from D2 would not have been lead to distinguishing feature (vi) in combination with the other claim features.

19. Conclusion

19.1 Hence, in view of the available prior art, the skilled person starting from either D1 or D2 would not have arrived at the method of claim 1 without inventive acti­vity. The subject-matter of claim 1 is thus considered to involve an inventive step, Articles 52(1) and 56 EPC.

19.2 This conclusion applies also to corresponding inde­pen­dent computer program claim 13 and device claim 14, and a fortiori to dependent claims 2 to 12 and 15.

Refund of further search fee

20. In a communication pursuant to Rule 64(1) EPC issued on 12 August 2014, including a partial European search report, the search division indicated that the original set of claims did not comply with the requirement of unity of invention, Article 82 EPC, identified three "search subjects":

(1) original claims 1-9, 12, 13, 15-17,

(2) original claims 10, 11,

(3) original claim 14,

and invited the applicant to pay two further search fees if the European search report was also to cover the "search subjects" (2) and (3) (Form 1507U in conjunction with the annex "sheet B" of the partial European search report).

21. In sheet B, the search division provided the following reasoning.

21.1 In sheet B, point 1, the search division stated that "a method for authenticating objects comprising the steps of claim 1 is known from D1 or D2", followed by an indication of passages of D1 and D2 for some features of claim 1. It was said that "in D1 the object instance must be recognized and its pose estimated before authentication [...] and it is trivial to postulate that this task should be carried out automatically", and that this applied also to D2. The search division then indicated that "a method for object recognition and pose estimation notorious in the art of machine vision to be used for this purpose is the bag of features method", and stated that it comprised "recog­ni­tion codebook training and its organization as a tf-idf file, object enrollment, object verification by comparing the features extracted from the current image with the codebook". For the "bag of features" method, reference was made to "online textbook: Szeliski - Computer Vision: Algorithms and Applications: Instance recognition p.690" but no corresponding document was cited in the partial search report. The search division concluded that "consequently, the subject-matter of claims 1-5 does not involve an inventive step (Article 56 EPC) with respect to D1 or D2".

21.2 In sheet B, point 2, the search division further indi­ca­ted that "with respect to the prior art, dependent claims 2-17 refer to potential special technical fea­tures which are grouped as follows: [...]", with an indication of features and - in all cases but the first one - solved problems for original claims (2-5), (6), (7, 8), (9, 12, 13), (10), (11), (14), (15-17). The following was indicated for original claims (2-5), (6), (10) and (11) (which are the only relevant ones for this decision):

"Claims 2-5 referring to various commonly known features of object recognition / authentication using a dictionary of features (BoF)";

"Claim 6 referring to the features of the choice of descriptors and/or geometrical relations between the keypoints solving the known problem of ensuring geometric consistency of multiple matched keypoints";

"Claim 10 referring to the feature of soft assignment in the quantisation step of codebook / feature dictionary generation by the function Rin(k) =Q(((xk )T (cn - c') - din /2)/sigma)";

"Claim 11 referring to the features of patch-wise alignment using local projection transforms solving the known problem of aligning the current object with a template object".

21.3 In sheet B, point 3, the search division concluded that because "the features of claim 1 are obvious", the aforementioned "groups of features" "solve different, juxtaposed problems" and there were "no common or corres­ponding technical features or effects in the two groups of claims" (meant was presumably: any two of these groups of claims), there was no single general inventive concept "relating the above groups". The application therefore lacked unity of invention, Article 82 EPC.

21.4 In sheet B, point 4, the search division indicated that original claims 2-9, 12, 13 and 15-17 were "grouped into one search subject", original claims 10 and 11 into another search subject, and original claim 14 appa­rently into a third one (and objections under Article 83 or 84 EPC were suggested in respect of that claim).

22. The appellant paid a further search fee only for original claims 10 and 11, i.e. the second "search subject".

23. The (final) European search report was issued on 31 October 2014, covering original claims 1-13 and 15-17. In the accompanying European search opinion (section I), the search division repeated the reasons given in sheet B of the partial European search report.

24. When entering the examination phase, the appellant requested a refund of the further search fee paid for original claims 10 and 11 (letter of 21 March 2016). The reason was that original claim 1 was inventive and its characterising part, which had not been considered in the search division's reasoning, represented the single general inventive concept shared by original claims 1 to 17. Moreover, it appeared that "questions of inventive step and unity of invention [had] been intermingled".

25. In the contested decision (point 7), the examining division confirmed the search division's finding of non-unity of the original set of claims and rejected the refund request.

25.1 The examining division maintained the search division's finding of lack of inventive step of original claim 1 simply by reference to the reasons it gave in the decision for claim 1 of the then main request, which was said to consist of "the subject-matter of original claim 1 further specified with features of original claim 3" (decision, point 7.1).

25.2 The remainder of the examining division's reasoning (decision, points 7.2 and 7.3) is practically verbatim that of the search division in sheet B, points 2-4 (see points 21.2 to 21.4 above), with the added remark that "the applicant's argument rests entirely on the allowability of claim 1" and that "the Examining Division does not share the view that original claim 1 is allowable and thus maintains the original objection of lack of unity under Article 82 EPC".

26. In the grounds of appeal (point IX.1), the appellant argued that "all [...] embodiments and options are related to the subject matter of claim 1 in unitary manner, given that the method of claim 1 uses both identification - and authentication features which might be in the form of claims 5 to 10, respectively of claims 11 to 14"; this applied to claim 1 according to the then main request and all auxiliary requests. The appellant also argued that "it isn't clear why and how the examining division combined claims 10 and 11 into a group and considered them as forming a (second) group of unitary inventions".

27. Rule 64(1) EPC provides that if a European patent application lacks unity, the applicant shall be informed that, for the European search report to cover any invention other than that first mentioned in the claims, a further search fee has to be paid.

Rule 64(2) EPC provides that any fee paid under Rule 64(1) EPC shall be refunded if, during the examination of the European patent application, the applicant requests a refund and the examining division finds that the communication under Rule 64(1) EPC was not justified.

28. As the appellant's refund request concerned only the further search fee paid for original claims 10 and 11, the examining division had to assess whether the communication under Rule 64(1) EPC was "justified" - within the meaning of Rule 64(2) EPC - to the extent that it required a further search fee to be paid for original claims 10 and 11.

29. The board considers it appropriate to first determine the scope of the review under Rule 64(2) EPC by the examining division (and by the board) before reviewing the reasoning provided by the examining division in the decision.

30. The scope of the review under Rule 64(2) EPC

30.1 In a number of board of appeal decisions, it has been considered that the scope of the review under Rule 64(2) EPC (previously Rule 46(1) EPC 1973) to be carried out by the examining division is limited to the consideration of certain facts and/or arguments.

30.1.1 Most notably, it was considered in T 188/00 that "a review of the finding of lack of unity of invention has to be carried out having regard only to the facts presented by the search division in its communication under Rule 46(1) EPC [1973]" and that "the examining division has to base its review solely on the documents cited in the partial search report and on the specification of the different inventions drawn up by the search division, while taking into account argu­ments which the applicant may have submitted in support of his request for a refund" (T 188/00, reasons 4.5).

30.1.2 That the review is limited to the specification of the different inventions drawn up by the search division is generally accepted in the case law and also by the present board, because any invitation to pay a further search fee under Rule 64(1) EPC must be for a specific invention or group of inventions that is to be covered by the search report and can thus only be justified for that particular invention or group of inventions.

30.1.3 The limitation to the facts presented by the search division, in particular the cited documents, postulated in T 188/00, has been followed, for instance, in T 1476/09, reasons 3, T 2285/17, reasons 4.2, and T 2873/19, reasons 9.1. In T 188/00 and in T 1476/09, the board found that the examining division's reasoning why the communication under Rule 64(1) EPC had been justified was incorrect because it relied on prior art that had not been cited in the partial search report (see T 188/00, reasons 4.6, and T 1476/09, reasons 3). In the cases underlying T 2285/17 and T 2873/19, the examining division's reasoning was limited to prior art cited in the partial search report and there was consequently no objection by the board in that respect.

30.1.4 In T 755/14, the board formulated a slightly different criterion. It considered that the review under Rule 64(2) EPC of a non-unity objection raised in the communication under Rule 64(1) EPC must be limited to the determination of whether that objection was justified at the time the communication was issued, and thus whether it was justified under consideration of the prior art that was available at that time (T 755/14, reasons 4). In that case, the board found that the communication under Rule 64(1) EPC had not been justified in view of documents not cited in the partial search report but only cited in the original application (T 755/14, reasons 5, 6.2 and 6.3).

30.1.5 Other decisions have gone beyond T 188/00 by stating that the review under Rule 64(2) EPC is to be based on the facts and arguments presented in the communication under Rule 64(1) EPC: see T 2526/17, reasons 4.7, and T 1414/18, reasons 4.3.

30.2 The present board agrees with T 188/00 to the extent that the examining division may only find that the communication pursuant to Rule 64(1) EPC was justified within the meaning of Rule 64(2) EPC on the basis of the facts regarding the prior art presented by the search division with that communication, in particular the documents cited in the partial search report including sheet B.

On the other hand, the present board considers that a finding that the communication was not justified may well be based on further facts (for instance in the circumstances of the case in T 755/14).

While the board essentially agrees with the position formulated in T 188/00, it is not fully convinced by the reasons provided for it in that decision, nor by the view expressed in T 2526/17 and T 1414/18 that the scope of the review should also be limited to the arguments presented by the search division, at least if understood as a limitation to only those arguments.

Instead, the present board considers that, in the context of Rule 64(2) EPC, the examining division may, in order to find the communication under Rule 64(1) EPC to have been justified, complete a reasoning outlined by the search division with the communication but may not replace it by an entirely different reasoning, even if based on the same prior art.

30.3 Limitation to facts

As regards the position formulated in T 188/00, the board notes the following.

30.3.1 In T 188/00, reasons 4.5, the board argued that this position followed essentially from the wording of Rule 46 EPC 1973 (now Rule 64 EPC). It also referred to the so-called "protest procedure" according to Rule 40.2(c) PCT before an ISA (and the similar procedure according to Rule 68.3(c) PCT before an IPEA) and noted that the boards of appeal had, in this context, ruled that the review of the justification for an invitation to pay additional fees had to be based exclusively on the reasons given in the invi­ta­tion having regard to the facts and arguments submitted by the applicants (reference was made to W 4/93, reasons 2.1 and 2.2).

The board also notes that essentially no additional reasons have been provided in the subsequent decisions cited above that follow T 188/00.

30.3.2 However, notwithstanding their similarities, Rule 64 EPC and Rule 40 PCT (and, a fortiori, Rule 68 PCT) have some notable differences. First, Rule 40.1(i) PCT expressly requires the invitation to pay additional fees to "specify the reasons for which the international application is not considered as complying with the requirements of unity of invention", which Rule 64(1) EPC does not. Secondly, in cases of a protest by the applicant, Rule 40.2(c) PCT foresees a "review body" that examines a "reasoned statement" filed by the appli­cant, decides whether and to what extent it finds the protest justified and orders corresponding reim­bursements, without entering into a dialogue with the applicant. Under Rule 64(2) EPC, the examining division is competent to decide on a refund request at any time during examination. Thirdly, Rule 40.2(c) PCT provides for a reimbursement of paid additional search fees to the extent that the review body "finds the protest justified", whereas Rule 64(2) EPC provides for a refund if "the examining division finds that the communication under [Rule 64(1) EPC] was not justified".

Furthermore, the relevant PCT Guidelines were binding guidance for the EPO when acting as an ISA and for the boards of appeal when they were deciding on protests against the charging of additional search fees under the PCT (G 1/89, headnote and reasons 6), so that several statements in the PCT Guidelines on how unity of invention is to be assessed and the protest proce­dure to be carried out have been taken over in the case law under the PCT already for that reason (see e.g. G 1/89, headnote, and W 21/03, reasons 4). Neither the PCT International Search and Examination Guidelines nor the Guidelines for Examination in the EPO are binding on the boards of appeal when reviewing decisions of examining divisions (Article 23(3) EPC).

30.3.3 Accordingly, the conclusions drawn in the case law for the review of invitations to pay additional search fees under Rule 40.2(c) PCT (or additional preliminary examination fees under Rule 68.3(c) PCT) do not immediately carry over to Rule 64(2) EPC. This applies, for instance, to the observation of the Enlarged Board of Appeal in G 1/89, reasons 8.2, that the review body under the PCT decides without the applicant having had an opportunity to comment. Under the EPC, Article 113(1) EPC ensures that the applicant's right to be heard is guaranteed even if the examining division goes beyond the reasons given by the search division.

30.3.4 As noted in G 1/91, reasons 4.1, the requirement of unity of invention ­under Article 82 EPC, notwithstanding its substantive nature, is "still merely an administrative regu­lation", serving a number of administrative purposes, an important one being financial. In conjunction with Rule 64 EPC, this requirement enables the patent office to collect an amount of fees dependent on the number of inventions or groups of inventions within the meaning of Article 82 EPC and thus approximately commensurate to the work it has to carry out (see also G 2/92, headnote).

30.3.5 In this context, the communication under Rule 64(1) EPC serves the purpose of allowing the search division to limit its initial search to the invention or group of unitary inventions first mentioned in the claims and to ask for a further search fee if the search is to be extended to cover an invention or a group of inventions that is non-unitary with that which has been searched so far.

An invitation to pay a further search fee may thus only have been "justified" in view of the prior art that was available to and considered relevant by the search division at the time the communication was issued.

The examining division may therefore only find that the communication under Rule 64(1) EPC was justified within the meaning of Rule 64(2) EPC on the basis of the facts regarding the prior art presented by the search division with that communication, in particular the documents cited in the partial search report including sheet B.

30.3.6 On the other hand, the present board considers that the examining division may well rely on further facts in a finding that the communication under Rule 64(1) EPC was not justified, for instance in the circumstances of the case in T 755/14 (see point 30.1.4 above) or, say, if further evidence shows that the only document cited in the partial search report and relied on in an a posteriori non-unity finding does actually not belong to the prior art.

30.3.7 Such further facts need also not have been submitted by the applicant. More generally, the board considers that the examining division may find that the communication under Rule 64(1) EPC is not justified for reasons other than those presented by the applicant. Rule 64(2) EPC does not require the refund request to be reasoned and provides that any fee paid shall be refunded if "the examining division finds that the communication under [Rule 64(1) EPC] was not justified", without further qualification (whereas under the PCT the reimbursement requires the protest to be justified - the protest having to include a "reasoned statement" to the effect that the international application complies with the requirement of unity of invention or that the amount of required additional fees is excessive, Rule 40.2(c) PCT).

30.4 Limitation to arguments

As regards the position expressed in T 2526/17 and T 1414/18 that the scope of the review should not only be limited to the facts but also to the arguments presented by the search division in the communication under Rule 64(1) EPC, the board notes the following.

30.4.1 Rule 64(1) EPC does not expressly require the communication to include a reasoning as to why the search division considers the application to lack unity of invention. Literally, Rule 64(1) EPC only requires the communication to include a specification of the inventions or groups of inventions which have been searched and of those for which further search fees need to be paid if they are to be covered by the search report (as otherwise the applicant would not know for what the further search fees are to be paid).

30.4.2 In the present case, the communication was issued on 12 August 2014. At that time, a partial European search report was not accompanied by a search opinion; only the (final) European search report, issued after the time limit for paying further search fees had expired, was accompanied by a search opinion. The Guidelines for Examination in the EPO, September 2013 edition ("EPO Guidelines 2013"), which were then in force, only required the search division to include "the reasoning behind the lack of unity" in the search opinion accompanying the European search report (EPO Guidelines 2013, B-VII, 1.2, and B-XI, 5). Still, it was common for communications under Rule 64(1) EPC to include some reasoning in sheet B of the partial European search report, as in the present case (see also e.g. the cases underlying J 24/96 or T 389/03), but not all such communications did (as also noted in T 756/14, reasons 4.1).

30.4.3 This is contrast to Rule 40.1(i) PCT which requires the invitation to pay additional search fees to specify the reasons for which the international application is not considered as complying with the requirement of unity of invention. This might provide justification under the PCT for limiting the scope of the review to the reasoning and thus to the facts and arguments provided in the invitation. In the case law on the PCT protest procedure, this has led to the conclusion that an additional search fee is already to be reimbursed if the reasoning presented in the invitation is insufficient, as such a deficiency may not subsequently be remedied (see e.g. W 4/85, headnote and reasons 3, W 11/93, reasons 2.5, 3 and 4, and W 9/07, reasons 2).

30.4.4 In the present board's view, a similarly strict limi­ta­tion of the scope of the review under Rule 64(2) EPC to the arguments provided in the communication cannot be justified where the provision of any arguments was at the discretion of the search division. A complete reasoning for the lack of unity of invention may only be required from the examining division if it finds the communication to be justified (Rule 111(2) EPC).

This position is also coherent with the separation of competences between search and examining division foreseen in the EPC, with the consequence that the same standard of reasoning may not be required from the search and the examining division.

30.4.5 The board considers however reasonable that, in the context of Rule 64(2) EPC, the examining division may complete a reasoning that has only been outlined by the search division with the communication under Rule 64(1) EPC but may not replace it by an entirely different reasoning, even if based on the same prior art, to find that the communication was justified.

30.4.6 This position takes account of the fact that the applicant has to decide whether or not to pay further search fees only on the basis of the information provided with the communication under Rule 64(1) EPC. Where the search division decides to provide arguments with the communication as to why a further search fee would have to be paid, this is to enable the applicant to understand at this stage why the invitation is justified. The applicant would inevitably rely on these arguments to assess whether it is actually entitled to a complete search for a single search fee and, if so, would pay the further search fee with the expectation that it will be refunded upon review. In the board's view, it would be undesirable as a matter of fairness that such a refund may later be refused by the examining division because the communication is considered to have been "justified" based on an entirely different reasoning than that contained in the communication.

The board considers reasonable to interpret Rule 64 EPC in such a manner favourable to the applicant in view of the merely administrative nature of this rule (see T 708/00, reasons 6, for similar considerations).

The board also notes that J 24/96, reasons 3, seems to have relied on a similar position as that formulated at point 30.4.5.

30.4.7 The board emphasises that these considerations regar­ding a qualified limitation to arguments presented by the search division concerns only the decision of the examining division on whether further search fees are to be refunded under Rule 64(2) EPC. When assessing whether the application documents on file comply with the requirement of Article 82 EPC, as a requirement for the grant of a patent, the examining division is not bound by the search division's opinion on unity of invention (T 178/84, headnote 1, T 631/97, reasons 3.6; see also J 3/09, headnote 1).

30.4.8 The board also notes that nowadays the invitation to pay further search fees and the partial European search report are systematically accompanied by a search opinion that includes the reasoning for the non-unity findings (EPO Guidelines, March 2024 edition, F-V, 4.1). This change of practice was introduced as a "new service" of the EPO entering into force on 1 April 2017 with the Notice of the European Patent Office dated 3 March 2017 concerning issuing a provisional opinion accompanying the partial search results, OJ EPO 2017, A20. In this notice, it was indicated that "[a]s with international applications (cf. Rule 40.1 PCT), for European patent applications too applicants will in the future [i.e. as from 1 April 2017] be systematically informed about the reasons for the non-unity findings together with the invitation to pay further search fees".

31. Restraint to be exercised in the context of Rule 64 EPC

31.1 As to the appellant's argument that the search division had "intermingled" the requirements of unity of invention and inventive step, the board notes that the search division and the examining division are allowed to make objections of lack of unity of invention a posteriori, i.e. with respect to the prior art (G 1/89, headnote, and T 87/88, reasons 2). The following should however be considered.

31.2 In view of the administrative nature of Article 82 EPC and Rule 64 EPC, and in line with several board of appeal decisions (see e.g. T 806/18, reasons 5.4), an objection of lack of unity should not be insisted upon on the basis of a narrow, literal or academic approach, in particular during search when the possible lack of unity does not necessitate a further search (as also noted in the EPO Guidelines 2024, F-IV, 2.2, and in the EPO Guidelines 2013, F-V, 8).

Using the words of G 1/89, reasons 8.2 (albeit in the context of the PCT), the present board considers that a search division in a communication under Rule 64(1) EPC (or an examining division when reviewing such a commu­nication pursuant to Rule 64(2) EPC) may raise an objection of lack of unity in clear cases but should exercise restraint in the assessment of novelty and inventive step when carrying out an a posteriori assessment of unity of invention and in borderline cases preferably refrain from considering an applica­tion not to comply with the requirement of unity of invention on such grounds.

31.3 More concretely, in the board's view, the restraint to be exercised by the search division in a communication pursuant to Rule 64(1) EPC (and by the examining division in the context of Rule 64(2) EPC) has at least the following aspects.

31.3.1 While findings of lack of unity a posteriori are in principle allowed, they must be raised with caution especially when they rest on an objection of inventive step, as questions of inventive step often constitute the major issue in substantive examination and may be controversial until the end of the examination and/or appeal proceedings.

31.3.2 The search division should refrain from raising formalistic objections based on a literal interpetation of the claims, because the assessment of unity of invention by the search division in the context of Rule 64 EPC only serves the purpose of determining whether a partial search report is to be issued (see also T 631/97, reasons 3.6), and the search according to Article 92 EPC is to be based on the basis of the claims "with due regard to the description and any drawings". Findings on novelty or inventive step of the search division in its communication under Rule 64(1) EPC should thus be robust, especially against foresee­able amendments and against clarity issues that the claims may have.

32. The examining division's reasoning on Rule 64(2) EPC

32.1 In the present case, the search division provided reasons in sheet B of the partial search report why the original set of claims lacked unity of invention a posteriori in view of D1 or D2, as starting point, combined with the allegedly notorious "bag of features" method. For that method, reference was made in sheet B to "online textbook: Szeliski - Computer Vision: Algorithms and Applications: Instance recognition p.690" but a corresponding document was neither cited in the listing of "documents considered to be relevant" of the partial search report nor sent to the applicant with the partial search report.

32.2 The examining division found that the communication under Rule 64(1) EPC was justified and refused therefore a refund of the further search fee paid for original claims 10 and 11 (decision, point 7).

32.3 The examining division noted that claim 1 of the then main request consisted of "the subject-matter of original claim 1 further specified with features of original claim 3" and simply referred to the reasons given for claim 1 the main request to conclude that original claim 1 was not inventive (decision, point 7.1). In doing so, the examining division relied on D1 or D2 combined with common general knowledge exemplified by D7 for the lack of inventive step of original claim 1.

32.4 The board considers that D7, which was only provided to the applicant with the examining division's communication dated 20 September 2016, corresponds to the bibliographical data indicated by the search division in sheet B, so that the examining division's reasoning was correctly only based on facts presented by the search division with the communication.

32.5 The board notes that the appellant complained in the first-instance proceedings (letter dated 21 March 2016, pages 2-3) that "the features of the characterizing part of claim 1 apparently have not been considered" in the search division's objection of lack of inventive step of original claim 1 when deciding on unity of invention.

The board agrees with the appellant that the reasoning provided by the search division in sheet B for the lack of unity a posteriori does not address all the features of claim 1 and is more an outline of an objection than a complete objection. While it was as such understandable, it would not have been sufficient for a "reasoned" decision within the meaning of Rule 111(2) EPC, but that is not an applicable standard for the communication under Rule 64(1) EPC. The incompleteness of the search division's reasoning in the present case does not by itself imply that the communication was not justified within the meaning of Rule 64(2) EPC.

Within the framework of Rule 64(2) EPC, the examining division was competent to complete the reasoning of lack of unity outlined by the search division, and did so at least in respect of the finding of lack of inventive step of original claim 1, as the reasons provided in the decision for claim 1 of the then main request (which are relied on in the reasons for Rule 64(2) EPC) followed the outline provided by the search division and addressed the claim features in sufficient detail.

32.6 However, for the reasons given at points 16.3 and 18.1 above, the board is not convinced by the reasons given by the examining division on why claim 1 of the then main request lacked an inventive step over D1 or D2 combined with D7. It follows that the board is also not convinced by the reasons given by the examining division in the context of Rule 64(2) EPC (and a fortiori by the search division) on why original claim 1 lacked an inventive step.

32.7 Furthermore, the remainder of the examining division's lack of unity reasoning (decision, point 7.2) is practically verbatim the same as that presented by the search division in sheet B, points 2 to 4 (see points 21.2 to 21.4 above), and the board considers both to be flawed.

32.7.1 Based on the finding that original claim 1 was not inventive, the examining division appears to have considered that each of the following groups of original claims,

- original claims 2-5,

- original claim 6,

- original claim 7 and 8,

- original claims 9, 12 and 13,

- original claim 10,

- original claim 11,

- original claim 14,

- original claim 15-17,

represented a group of inventions with its own "potential special technical features" and they were such that no two of them were linked with any of the others by a single general inventive concept within the meaning of Article 82 EPC.

32.7.2 The grouping of original claims 2-9, 12-13 and 15-17 into a first "search subject" and of original claims 10 and 11 into a second "search subject" was "for the search". The board assumes that this was done because searching together the claims in one of these "search subjects" did require or would have required only negligible additional work (see EPO Guidelines 2013, B-VII, 2.2).

32.7.3 Ultimately, the invitation to pay a further search fee for original claims 10 and 11 (the second "search subject") may only have been justified if these claims were non-unitary with original claims 2-9, 12-13 and 15-17 (the first "search subject").

32.7.4 The reasoning of the examining division, like that of the search division, ignores at this stage the depen­den­cies of these claims (apart from their dependency on claim 1) and appears to have only considered the additional features defined in each of the claims.

32.7.5 Original claim 10 was directed to a "method according to one of the preceding claims 3 to 9" and original claim 11 to a "method according to one of the preceding claims 3 to 10".

Hence, at least the subject-matter of original claim 3 (itself being dependent on original claim 1) was common to the first and the second "search subjects".

They may only be non-unitary if original claim 3 was not inventive, but this has neither been alleged nor established by the examining division in its reasoning on Rule 64(2) EPC.

The board notes that claim 1 of the then main request, to which the examining division referred, did include some features of original claim 3 but not all of them, for instance the step between identification and authentication of aligning the authentication features of the acquired image (essentially feature 5.4).

32.7.6 Furthermore, the examining division did not clearly spe­cify the special technical features within the meaning of Rule 44(1) EPC that characterise the first "search subject". Original claims 2-5 were said to involve "potential special features" but these were identified as "commonly known features of object recognition / authentication using a dictionary of features (BoF)", which rather suggests that these features were considered to be also not inventive over the prior art. Original claim 6 is the first claim in the first "search subject" for which more concrete "potential special features" are identified: "features of the choice of descriptors and/or geometrical relations between the keypoints solving the known problem of ensuring geometric consistency of multiple matched keypoints" (decision, point 7.2).

To be consistent with this finding, the examining division should have concluded that at least the alternatives of original claims 10 and 11 that are dependent on original claim 6 - which have been included in the second search subject - comprise the same special technical features as the first search subject, so that there is no lack of unity between the first and the second search subjects, Article 82 EPC and Rule 44(1) EPC. The examining division's reasoning is thus not consistent.

32.8 It follows that the board is not convinced by the examining division's reasoning as to why the communication pursuant to Rule 64(1) EPC was justified.

32.9 The board may now in principle either remit the case for further prosecution or exercise the power within the competence of the examining division, Article 111(1) EPC (see however T 188/00, reasons 4.6, where the board decided to immediately refund the further search fee without further reasoning after having determined that the decision of the examining division under Rule 64(2) EPC was incorrect). The board does the latter.

32.9.1 Having regard to the facts and arguments presented by the search division in the communication under Rule 64(1) EPC, the communication could only be justified if - as a minimum - original claim 3 were not inventive in view of D1 or D2 combined with D7.

32.9.2 The board does not see how such an objection of lack of inventive step on this factual basis could succeed. The objection presented by the examining division in the decision has not convinced the board. The line of argument suggested at point 18.5 above is based on D2 combined with D8, but the latter was not cited in the partial search report and cannot thus be considered in the context of Rule 64(2) EPC. Furthermore, this tentative line of argument would also fail to show that the skilled person would have arrived at the alignment step between the identification and the authentication, which is a feature of original claim 3.

32.9.3 The board has also considered whether the fact that original claim 3 fails to explicitly specify that the result of certain steps (e.g. the generation of the codebooks) are actually used in the remainder of the claim could justify disregarding the steps from the inventive step analysis (because they would as a result not contribute to a technical effect). But any objection of lack of inventive step of this kind - relying critically on clarity issues with original claim 3 - would not be "robust" in the sense of point 31.3.2 above and would also be too complex to be suitable to support an objection of lack of unity a posteriori in the context of Rule 64 EPC.

32.9.4 The board also notes that the objection of lack of inventive step outlined by the search division in the communication and its further development by the examining division in the decision were arguably already too complex and controversial to be suitable to support an objection of lack of unity of invention a posteriori in the context of Rule 64 EPC.

33. The board therefore concludes that the communication pursuant to Rule 64(1) EPC was not justified and the further search fee paid for original claims 10 and 11 is thus to be refunded, Rule 64(2) EPC.

Reimbursement of appeal fee

34. The appellant alleged that substantial procedural violations occurred in the first-instance proceedings.

34.1 It submitted that the conduct of the examination proceedings represented an undue burden to it, which could have been prevented if the examining division "had applied already at the initial stage of the preliminary opinion on patentability the objective methodology of evaluation typically used by the EPO and set out in detail in the EPO's [Guidelines] or would have applied such methodology during the further course of the proceedings" (grounds of appeal, section IX.2, page 25, second full paragraph).

Until the decision, the examining division "didn't provide sufficiently detailed information on the objections, respectively contained objections in generalized manner, without taking into account the facts and arguments provided by the appellant, such that the appellant de facto was deprived of first instance proceedings". Such conduct of the examination proceedings did not respect the appellant's right to be heard according to Article 113(1) EPC (see appellant's letter dated 3 December 2023, page 3, first paragraph).

34.2 Moreover, the appellant referred to a telephone call from the first examiner on Friday, 25 May 2018, i.e. 1.5 working days before the day of the scheduled oral proceedings on Tuesday, 29 May 2018, during which the appellant was informed that the examining division could not take a decision according to the state of the file, as requested by the appellant, and that the oral proceedings would take place as scheduled. On this occasion, the appellant expressed its surprise that the examining division did not become aware of this fact until 1.5 working days ahead of the oral proceedings and reiterated its request to continue the written procedure.

34.3 The appellant further alleged that in a subsequent telephone call with the chairman of the examining division on the same day, the chairman had stated that he "blindly trust[ed]" in his first examiner and that he did not even know who the applicant was in this file. The appellant considered that in view of the volume and complexity of the file, it had been simply impossible for the chairman and any other member of the examining division to get sufficiently acquainted with the matter within 1.5 working days and to build an independent opinion. Against this background, it was unclear how the examining division had assured "an objective decision making during the oral proceedings".

35. Rule 103(1)(a) EPC provides that the appeal fee should be reimbursed in full where the board of appeal deems an appeal to be allowable, if the reimbursement is equitable by reason of a substantial procedural violation.

36. A wrong assessment of prior art or technical content constitutes an error of judgment, i.e. a substantive error, and not a procedural violation, let alone a substantial one (see also the Case Law of the Boards of Appeal of the European Patent Office, 10th edition 2022, V.A.11.6.10 b), with further references cited there).

37. In the board's view, the appellant's allegation that the examining division did not apply "the objective methodology of evaluation typically used by the EPO" (see point 34.1 above) merely criticises a wrong assessment of prior art or technical content and therefore to an error of judgment. This does not, however, constitute a procedural violation justifying the reimbursement of the appeal fee.

38. The board further considers that the objections raised by the examining division in its communication, albeit possibly flawed, were sufficiently detailed for them to be addressed by the appellant. Also the appellant did not argue that it had not been given an opportunity to comment on the reasons relied upon in the contested decision. The decision contains a section addressing in detail the main arguments provided by the appellant during the examination proceedings (decision, point 1.3). Hence, the board considers that the appellant's right to be heard, Article 113(1) EPC, has not been infringed.

39. Furthermore, the board fails to see a procedural viola­tion in the conduct of the proceedings by the examining division, i.e. in its decision not to continue the written procedure and not to take a decision "according to the state of the file" but to hold the scheduled oral proceedings in the absence of the appellant. The examining division has discretion to hold oral procee­dings under Article 116(1) EPC, if it considers oral proceedings expedient. The fact that the first examiner did not inform the appellant explicitly about the maintenance of the oral proceedings until 1.5 working days before the scheduled date of the oral proceedings (see point 34.2 above) is of no relevance in this respect. The examining division was not obliged to call the appellant to inform it about the maintenance of the oral proceedings; the fact that they were not cancelled was sufficient information in this regard.

40. Lastly, regarding the alleged statements of the chair­man during the conversation with the appellant (see point 34.3 above), the board notes that they were not included in the minutes of the telephone conversation dated 28 May 2018 and the appellant did not request a correction of the minutes. Hence, the board cannot take this alleged fact into account. But even assuming, for the sake of argument, that statements to that effect were made by the chairman, the board fails to see any robust evidence for the circumstance insinuated by the appellant that the members of the examining division were indeed ill-prepared.

41. As it is not apparent that any substantial procedural violation occurred in the first-instance proceedings, the appeal fee is not to be reimbursed, Rule 103(1)(a) EPC.

Order

For these reasons it is decided that:

1. The decision under appeal is set aside.

2. The case is remitted to the examining division with the order to grant a patent in the following version:

Description:

Pages 1/42 to 42/42 according to the new main request received during oral proceedings of 6 December 2023

Claims:

No. 1 to 15 according to the new main request received during oral proceedings of 6 December 2023

Drawings:

Sheets 1/15 to 15/15 as originally filed.

3. One further search fee is refunded.

4. The request for reimbursement of the appeal fee is refused.