“description” - description of the experiment to run.
“tasks”- listofassessments.
“query_title” - the title of the query to be displayed.
“setting” - an extra requirement that the user should take into account when doing the assessment.
“ranking_type” - the order in which to display the ranking of the documents. It should match the“ranking_type” defined in the configuration file. “ranking_type” could have the following values:
“original” - sorted by the “score”.
“ranker_” - sorted by the predicted score of a pre-trained model.
“pre/in/postprocessing_” - displays the ranking generated by the fairness intervention.
Experiment File for the Score Annotation UI
“exp_id” - id of the experiment.
“description” - description of the experiment to run.
“tasks”- listofassessments.
“query_title” - the title of the query to be displayed.
“setting” - an extra requirement that the user should take into account when doing the assessment.
“index” - the index of the item to be annotated.
Experiment File for the Ranking Comparison UI
“exp_id” - id of the experiment.
“description” - description of the experiment to run.
“tasks” - list of assessments.
“query_title” - the title of the query to be displayed.
“ranking_type_1” - the order in which to display the ranking of the documents that is displayed on the left side. It should match the “ranking_type” defined in the configuration file.
“ranking_type_2” - the order in which to display the ranking of the documents that is displayed on the right side. It should match the “ranking_type” defined in the configuration file.
“ranking_type” could have the following values:
“original” - sorted by the “score”.
“ranker_” - sorted by the predicted score of a pre-trained model.
“pre/in/postprocessing_” - displays the ranking generated by the fairness intervention.