Statement Relevance Evaluation Guidelines

First of all, we would like to thank you for taking part in our evaluation.

The goal of this evaluation is to assess the relevance of statements with respect to ontologies.

What we mean by relevance: In the context of this evaluation, a statement is considered relevant w.r.t. an ontology, if it's important to be added to the ontology. For example, the statement Conference--subClassOf--> Event, is relevant to an ontology describing an academic domain, while Death--subClassOf-->Event, is probably irrelevant.

In this evaluation, you will be given an ontology (which is visualized in 2 different layouts), and 100 statements that you need to decide (by choosing the corresponding answer on the right of the interface as per figure below) whether each is:

• Relevant: The statement is important to add to the ontology.
• Irrelevant: It is not important to add to the ontology. In this case whether the statement is incorrect by itself (e.g. Car--subClassOf-->Person), or it does not fit in the context of the ontology (e.g. Death--subClassOf-->Event is correct by itself, but does not fit in e.g. an ontology in the academic domain).
• Don't know: There are cases where you are unable to tell if the statement is relevant or not, and please give the reason of not knowing.

Quick instructions:

• In order for the evaluation to go smoothly, consider yourself extending the given ontology visualized on the left of the interface (as per figure below), and that you are being suggested statements to decide whether to add or not.
• If for some reason you have to pause the evaluation, you can always go the initial link emailed to you, and the sytem will automatically resume at the last statement you've reached.
• You can download the OWL file of the ontology by pressing on the "Download OWL File" link in the interface (top-left of the figure below) if you prefer to open it in an ontology editor.

If you got any question, please don't hesitate to contact Fouad.

Thank you!