T 2087/15 () of 21.7.2020

Summary of Facts and Submissions

I. The appeal is against the decision of the examining division refusing the European patent application No. 07 834 775.4 on the the ground of lack of inventive step within the meaning of Article 56 EPC 1973.

II. In a communication, the Board indicated its provisional opinion that the claimed invention lacked an inventive step. With its response dated 21 April 2020, the appellant filed a amended set of claims.

III. The appellant requests that the decision under appeal be set aside and a European patent be granted based on its request filed on 21 April 2020.

IV. Claims 1 and 8 according to the sole request correspond in substance to claims 1 and 8 underlying the impugned decision, except for minor amendments of editorial nature made in claim 1 (see below, emphasis added by the Board).

Claim 1 is worded as follows:

System for generating exams, questionnaires or similar skill-level exams administered in large numbers, including a set of questions to be answered,

the system comprising a data base including a number of questions in at least one topic, each question being associated with a data set related to answers given to the questions in previous uses of the questions in question sets presented to at least one exam candidate, the database also comprising statistical data related to calculated facility, selectivity and distribution for said questions based on said previous answers, representing a measure of the quality of a set of questions,

said database is stored on at least one data storage device,

the system further comprising computer-based selection means for randomly selecting a question within one or more predetermined topics from said data base,

and a computer-based evaluation means for evaluating for possible inclusion in a question set for presentation to exam candidates the selected questions relative to predetermined requirements to the selected questions and said measure of the quality of said question[deleted: s] set,

and discarding from possible inclusion questions not fulfilling said requirements thereby to generate from said database a question set for presentation to exam candidates, wherein selected questions that do not fit a chosen chosen [sic] measure regarding selectivity and facility are rejected from the new question set, such that a controlled distribution of difficulty is obtained

wherein the evaluation means, based on said measures is adapted to reject selected questions not fulfilling the predetermined requirements by fitting a number of correct answers per failure in the selected questions with predetermined distributions, so as to ensure the controlled distribution of difficulty in the new question set; and

wherein the evaluation means is further adapted to indicate a quality of the new question set by comparing an actual reliability of the new question set in use to a calculated reliability, assuming the new question set to have a same distribution of characteristics as a known set of [deleted: of] the questions having known reliability.

Claim 8 has the following wording:

Method for generating exams, questionnaires or similar skill-level tests administered in large numbers, the method comprising the steps of:

• storing a number of questions in at least one topic a data base on [sic], each question being associated with a data set related to answers given to the questions in previous exams, said database being stored on at least one data storage device,

• randomly selecting by computer-based means a question within one or more predetermined topics from said data base,

• evaluating by computer-based evaluation means the selected questions relative to predetermined requirements related to a measure based on facility, selectivity and distribution corresponding to said data set for possible inclusion in a question set for presentation to exam candidates and discarding questions not fulfilling said requirements thereby generating a question set for presentation to exam candidates.

V. The appellant's arguments provided in the statement of grounds of appeal can be summarised as follows:

The position of the examining division that the effect of the invention was not an objective technical problem, but only took place in the mind of students facing difficult questions was not correct, because the invention took into account the correlation with answers to other questions to improve the quality of each question. Neither the question generation nor the answers were per se part of the invention. The invention used both questions and answers to improve a generated exam allowing some questions to be removed from the set based on the analysis, leaving only good performing questions to be used for new tests and exams. This was obtained by analysing the relationship between the questions and the provided answers, and not by considering whether or nor the answers were correct.

Reasons for the Decision

1. The appeal is admissible.

2. Procedural matters

In a communication pursuant to Rule 100(2) EPC, the Board informed the appellant about its provisional opinion that claims 1 and 8 lacked an inventive step (Article 56 EPC 1973).

In its reply dated 21 April 2020, the appellant provided further arguments and filed a set of claims 1 - 14 corresponding substantively to the set of claims underlying the contested decision, with the exception of minor amendments of editorial nature made to claim 1. The appellant further referred to a "discussion of the communication of the Board of Appeal from the inventor" and submitted a document containing arguments from one of the inventors as an annex to its letter.

Oral proceedings were not requested.

In accordance with Article 12(8) RPBA 2020, the Board concluded that the case was ready for decision.

3. The invention

The present invention relates to a method and system for improving the quality of computer generated sets of examination questions utilizing empirical data of question items from previous exams. The data are stored in a data base of a data storage device and can include e.g. facility/difficulty, selectivity, distribution of answers (page 5, lines 7 - 9), the correlation between the quality of the answers given to the question and the quality of the rest of the exam given by a person (page 4, lines 7 - 12), the deviation in the responses given to the question (page 4, lines 12 - 14), or other criteria (page 4, lines 16 - 17). The method comprises a step of selecting a random question from the database and evaluating the selected question based on predetermined requirements for the possible inclusion in a question set (e.g. an exam or a questionnaire) for presentation to exam candidates. Questions found not fulfilling the requirements are discarded so that a question set of better quality is obtained. The steps of storing data, selecting, evaluating and possibly discarding question items are performed by computer-based means (see page 7, lines 26 - 27).

4. Independent method claim 8

4.1 The scope of method claim 8 is broader than the method implemented on the system according to claim 1, because the method according to claim 8 lacks inter alia a step of indicating "a quality of the new question set by comparing an actual reliability of the new question set in use to a calculated reliability, assuming ..." according to claim 1, last four lines.

Therefore, the Board focuses on method claim 8.

4.2 The Board considers the technical features of claim 8 to be a "data storage device", a "data base stored on the data storage device", "computer-based means", "computer-based evaluation means". In other words, the technical features of claim 8 correspond to a standard computer having a memory ("data storage device") and a processor ("computer-based means", "computer based evaluation means").

While the Board accepts that claim 8, being a method involving technical means, has an overall technical character (see e.g. T 258/03, Headnote I), it comprises a mixture of technical and non-technical features.

4.3 According to the method of claim 8, it is the aim of the present invention to generate exams, questionnaires or similar skill-level tests administered in large numbers, by:

a) randomly selecting a question within one or more predetermined topics from a number of existing questions in at least one topic, each question of said number of existing questions being associated with a data set related to answers given to the questions in previous exams,

b) evaluating the selected questions relative to predetermined requirements related to a measure based on facility, selectivity and distribution corresponding to said data set for possible inclusion in a question set for presentation to exam candidates and

c) discarding questions not fulfilling said requirements thereby generating a question set for presentation to exam candidates.

The Board is of the opinion that steps a) to c) are well-known tasks of people preparing exams (e.g. teachers) who for centuries have performed this task mentally or on paper. A school teacher or an instructor preparing an exam e.g. for driving test candidates or for pilots selects question items from a pool of existing questions used in previous tests and evaluates these items with respect to their facility (i.e. how well or badly the question was answered in previous exams), selectivity (i.e. how representative the question is for the ability the exam aims to test) or distribution (i.e. how the scores in previous exams were distributed among the different candidates). If they find the selected and evaluated question unsuitable for the test (e.g. because it was too easy or too difficult for the average candidate in previous tests or because it revealed being erroneous or unclear), they will discard this particular question and not include it into the test they are preparing. However, if they find the question item suitable for the test, they will consider its inclusion in the test.

It follows that steps a) to c) for adapting questions to the level of students are not related to solving a technical problem, but to the non-technical task of a test writer (e.g. a teacher).

Even when characterised quantitatively, the effects underlying the claimed selection criteria are non-technical effects, merely possibly taking place in the minds of students facing easy or difficult questions, as they might face a test which corresponds more or less to their individual intellectual abilities. The same effects are expected to be achieved by the question sets resulting from the above mentioned steps a) to c).

The appellant provided arguments with respect to the technical/non-technical features of claim 8, which have not convinced the Board for the reasons as follows:

4.3.1 The Board is of the opinion that the appellant's argument that the invention took into account the correlation with answers to other questions to improve the quality of each question - see statement of grounds of appeal, page 1, third and fourth paragraphs - is not relevant, because in the method according to claim 8 a step of evaluating the "correlation" of a given question item with answers to other questions is not a part of the claimed method. This appears to be confirmed by the appellant's statement in the annex to the letter dated 21 April 2020, page 9, first paragraph.

Even if the Board were to accept that the invention took into account the correlation with answers to other questions to improve the quality of each question, the effect of this feature is not of technical nature, but would only improve a teaching method or competence screening, which is equivalent to an improvement of a non-technical activities, essentially being based on mental acts.

4.3.2 The Board does not share the appellant's view that "the present invention would work without human interaction" so that it "does not necessitate that any part would be performed in the mind of a student, even if this is the preferred use", see the statement of grounds of appeal, page 1, last paragraph, page 2, first paragraph. The method according to claim 8 uses a data base comprising data sets "related to answers given to the questions in previous exams" and aims at "generating a question set for presentation to exam candidates". A skilled person understands that the term "previous exams" does not relate to exam performed by a computer, but by human candidates so that the data sets are the result of human interaction, namely the performance of human test candidates in past exams. Moreover, the skilled person understands that the questions sets generated are to be presented to human exam candidates so that it can reasonably be argued that those effects provided by steps a) to c) which are actually sought to be achieved by the invention and in the end might provide an objectively measurable difference over prior art question sets - i.e. that the question sets generated perform better than previous question sets - might possibly take place in the minds of these candidates.

4.3.3 The argument of the appellant that the questions were not generated by a machine, see statement of grounds of appeal, page 2, first and second paragraph, is not relevant, because neither the examining division in their decision nor the Board understand the invention in a way that the question items themselves inevitably would be machine generated, and there is no indication in the application as filed that this would be the case.

4.3.4 The appellant pointed out that the invention did not relate to the generation of individual questions, but how it handled existing questions, "which may or may not have been used in previous exams".

The Board shares this view insofar the claimed method is not related to the generation of questions or answers, but to the generation of a set of questions for future presentation to exam candidates. It is clear to the Board that the origin of the questions themselves is left open, and the questions may obviously be human generated (but in theory also computer generated). The Board however observes that claim 8 is limited to questions that have been used in previous exams, because according to claim 8, steps a) to c) concern questions having a data set in the data base so that they necessarily have been presented in previous exams.

4.3.5 In the statement annexed to the letter dated 21 April 2020, page 2, part B), the appellant argued that the invention "consistently" improved "the quality of a question data base, and hence the quality of questionnaires produced from that" in the case of multiple choice question (MCQ) exams. In part C) on page 2 of the same statement, the appellant questioned if steps a) to c) had been performed by teachers for "centuries". The appellant also stated that the invention did "not aim to improve the content quality of each question".

The Board is not convinced by these arguments, because claim 8 neither comprises any means for improving the quality of the database nor is it limited to MCQ exams. However, even if the Board were to accept that the quality of the questions in a data base would improve, this cannot be qualified as a technical effect. The Board is of the opinion that steps a) to c) are performed by a person preparing an exam with questions to be presented to candidates. These type of exams existed well before the priority date of the present application, e.g. for the Chinese imperial examinations (606 - 1905). Hence, the Board agrees with the appellant that the claimed method does not change an individual question (i.e. does not improve its quality), but aim at providing an improved set of questions for an exam.

4.3.6 The appellant stated in the last paragraph of the statements of grounds of appeal that the technical character of the invention would be "related both to the objectivity and efficiency of the system as well as the fact that a manual operation handled by humans simply is not possible". The question sets were produced as a data selection resulting from statistical analysis of the registered responses to the questions. In the statement annexed to the letter dated 21 April 2020, page 2, part A), the appellant also argued that the invention was "completely separated from the subjectivity and illusiveness of what goes on in the mind of students and teachers". On page 3 of the same statement, the appellant argued that the invention is not an integrated part of the so-called "teaching sphere".

The Board is not convinced by these arguments, because claim 8 is not limited to a "large amount of input" difficult to handle by a human operator (e.g. a teacher). It might be correct that a test writer would have difficulties to mentally handle fifty thousand questions. In the Board's view there would be no problem to handle an ensemble of e.g. fifty questions known to the test writer and used in previous tests. As claim 8 does not specify the number of questions from previous exams in the data base, the method involving steps a) to c) on a computer is not necessarily more efficient. Furthermore, if the decision of including or discarding is based e.g. on the difficulty of the selected question (i.e. the number of correct answers compared to the number of total answers), it is not plausible that the method implemented on a computer would be necessarily more objective (or less subjective) than the same method performed by a human. Moreover, claim 8 does neither exclude any additional "manual operation" by a human being, nor does it require any statistical analysis, "sound mathematics and statistical procedures" or "special statistical arrangements" as argued in the statement annexed to the letter dated 21 April 2020. In other words, the Board is not convinced that the claimed method is necessarily performed more efficiently and/or less subjectively by computer-based means, compared to a human test writer. Finally, the Board cannot agree that the steps a) to c) could not be a part of what the appellant calls "teaching sphere", because the function of exams e.g. provided by a teacher to his students is to evaluate if his teaching was successful. Hence, it cannot be said that steps a) to c) are outside the "teaching sphere".

4.4 Therefore, the Board shares the view of the examining division that the technical features of the independent method claim 8 are not more than a computer having a data storage device (i.e. a memory) for storing a data base and a processor for implementing a method involving non-technical method steps a), b) and c) on this computer.

4.5 It is established case law that non-technical features cannot contribute to inventive step. Therefore, non-technical features may legitimately be part of the problem to be solved (T 641/00), for example in the form of a requirement specification given to the skilled person to implement, see Case Law of the Boards of appeal, 9th edition, 2019, I.D.9.1.4.

Thus, the sole technical problem derivable from the wording of claim 8 would be the proper configuration and programming of known technical means (i.e. a computer having a data storage device) in order to implement non-technical (teaching) constraints, i.e. the method steps a) to c).

The objective technical problem to be solved would thus be formulated as how the skilled person (a software programmer expert) would implement steps a) to c) of section 4.3 above in a standard computer.

The appellant provided arguments with respect to the objective technical problem and the relevant skilled person, which have not convinced the Board for the reasons as follows:

4.5.1 The appellant argued in its letter dated 21 April 2020 that the relevant person skilled in the art was not a "programmer", but a "person who aims at making as objective exams as possible without relying on the input from the lecturer from the moment the question has been made". A programmer "would neither be aware of the problem nor knew about the possibility to provide and use the information about the question set". The Board may accept this argument concerning the skilled person, however it appears questionable if the skilled person defined by the appellant is inevitably a technically skilled person, apart from the fact that it does not appear to correspond to any usual professional. It still mostly appear to correspond to a teacher in general, and therefore his contribution to the problem solved is to be taken as being given for the purposes of the problem-solution approach, as explained above in point 4.5.

It is well established case law, that the person skilled in the art in the case of computer-implemented inventions is an expert in a technical field, see Case Law of the Boards of Appeal, 9th edition, 2019, I.D.8.1.4. For example, if the technical problem concerns a computer implementation of a business, actuarial or accountancy system, it will be a person skilled in data processing, and not merely a businessman, actuary or accountant (T 172/03). In the present case, the objective technical problem lies in the implementation of a non-technical method on a computer so that the skilled person is a computer engineer, and not a person specialized in creating exams (e.g. a teacher). As explained in section 4.5 above, the computer engineer would have steps a) to c) as requirement specifications given to implement on a standard computer.

4.5.2 The appellant argued in the letter dated 21 April 2020 that the neither a programmer nor a teacher would perform step b) as the teacher would rely on his understanding of the field and how to make and respond to the individual questions. They would not take into account the "statistical information about the question sets" and reject a question "based on the statistics". The appellant stated that no proof was provided that the teacher would apply statistical methods.

The Board notes that claim 8 does not include any step of performing a statistical analysis to decide whether to reject a selected question or not. No statistical method is mentioned. Claim 8 only requires an evaluation of the selected question based on e.g. its difficulty. In the Boards view, a test writer (e.g. a teacher) would take into account the difficulty of a selected question based on the answers in previous exams before deciding to include it into a question set to be presented in a future exam. Hence, a programmer would be provided with a method including also step b) as requirement specification.

4.5.3 It may be understandable that this settled approach of the case law - namely that the fictional skilled person is already given elements of the inventive solution - may appear as obviously wrong in view of the present invention. However, as explained in T 1543/06, Reasons 2.6, the fiction of using an artificial (or hidden) starting point must be accepted as "an artefact of the systematic use of the problem-solution approach for assessing inventive step" for inventions involving both excluded and non-excluded subject-matter.

4.5.4 In his letter dated 21 April 2020 the appellant discusses the possibility of including "new" questions (i.e. those that have never been presented in an exam) into a question set, see page 3, third paragraph or the paragraph bridging pages 2 and 3, see also the statement annexed to the letter dated 21 April 2020, page 9, fifth paragraph and page 11, last paragraph. These arguments are not relevant for claim 8, because the claimed method comprises evaluating selected questions having data in the database, i.e. questions that have already been presented in previous exams.

4.6 Therefore, for the Board, the objective technical problem is the one given in section 4.5 above.

4.7 The Board points out that the application itself does not provide any specific way of how the objective technical problem is to be solved, but merely states on page 7, lines 26 - 27 that "all of these may be programmed into a computer or a computer network using general programming tools". The Board is of the opinion that a skilled person (i.e. a computer specialist) being provided with the above requirement specifications, i.e. steps a) to c), would implement them in a straightforward manner as part of his daily routine, that is to say without making an inventive step within the meaning of Article 56 EPC 1973.

When doing so, the skilled person would store the questions as well as the related data set associated thereto on a data storage device and perform steps a) to c) by computer-based means. In other words, it would arrive at a system comprising data storage means and computer-based means for performing steps a) to c).

Therefore, the subject-matter of claim 8 does not include an inventive step (Article 56 EPC 1973).

5. The sole request does not meet the requirements of Article 56 EPC 1973. Hence, the appeal must fail.

Although the Board refrains from deciding on the system according to claim 1, it is noted that the Board's communication stated reasons why the subject matter of claim 1 appeared to lack an inventive step as well.

Order

For these reasons it is decided that:

The appeal is dismissed.