BioASQ Participants Area

Task 2b: Test Results of Phase B

The test results are presented in seperate tables for each type of annotation. The "System Description" of each system is used.
The evaluation measures that are used in Task B are presented here .

Test batch 1

Exact Answers

	Yes/No	Factoid			List
System	Accuracy	Strict Acc.	Lenient Acc.	MRR	Mean precision	Recall	F-Measure
main system	0.5938	0.0400	0.1600	0.1000	-	-	-
Biomedical Text Ming	0.9375	0.1600	0.1600	0.1600	0.0572	0.0702	0.0614
SNUMedinfo1	-	-	-	-	-	-	-
SNUMedinfo2	-	-	-	-	-	-	-
SNUMedinfo3	-	-	-	-	-	-	-
SNUMedinfo4	-	-	-	-	-	-	-
SNUMedinfo5	-	-	-	-	-	-	-
system 2	0.9375	0.0400	0.1600	0.1000	-	-	-
Wishart-S3	0.8438	0.4400	0.4800	0.4600	0.4478	0.3335	0.3456
Wishart-S2	0.8438	0.4400	0.4800	0.4600	0.4774	0.3335	0.3621
system 3	0.9375	0.0400	0.1600	0.1000	-	-	-
BioASQ_Baseline	0.5313	-	-	-	0.0418	0.0766	0.0501
BioASQ Baseline FS	0.5000	-	-	-	0.0418	0.0766	0.0501

Ideal Answers

	Automatic scores		Manual scores
System	Rouge-2	Rouge-SU4	Readability	Recall	Precision	Repetition
main system	0.4387	0.4538	3.29	4.30	3.28	3.55
Biomedical Text Ming	0.1724	0.1823	4.04	3.24	3.72	4.57
SNUMedinfo1	0.4065	0.4100	4.06	4.13	3.75	4.08
SNUMedinfo2	0.3778	0.3793	4.02	3.99	3.79	4.06
SNUMedinfo3	0.4971	0.4971	3.88	4.29	3.56	3.61
SNUMedinfo4	0.3529	0.3542	3.98	3.97	3.74	3.96
SNUMedinfo5	0.4602	0.4679	3.68	4.19	3.54	3.45
system 2	0.4400	0.4544	1.65	2.16	1.62	1.86
Wishart-S3	0.4802	0.4814	-	-	-	-
Wishart-S2	0.4802	0.4814	3.75	4.22	3.52	3.59
system 3	0.4395	0.4535	1.43	1.99	1.52	1.69
BioASQ_Baseline	0.3741	0.3943	-	-	-	-
BioASQ Baseline FS	0.3622	0.3970	-	-	-	-

Test batch 2

Exact Answers

	Yes/No	Factoid			List
System	Accuracy	Strict Acc.	Lenient Acc.	MRR	Mean precision	Recall	F-Measure
Biomedical Text Ming	0.8214	-	-	-	0.1596	0.2057	0.1618
main system	0.8214	0.0435	0.1739	0.0942	-	-	-
SNUMedinfo2	-	-	-	-	-	-	-
SNUMedinfo1	-	-	-	-	-	-	-
SNUMedinfo3	-	-	-	-	-	-	-
SNUMedinfo4	-	-	-	-	-	-	-
SNUMedinfo5	-	-	-	-	-	-	-
system 2	0.8214	0.0435	0.1739	0.0942	-	-	-
system 3	0.8214	0.0435	0.1739	0.0942	-	-	-
system 4	0.7500	0.0435	0.1739	0.0942	-	-	-
Wishart-S1	0.9286	0.1304	0.1304	0.1304	0.4396	0.4739	0.4122
Wishart-S2	0.9286	0.1304	0.1304	0.1304	0.5120	0.4399	0.4261
system 5	0.7143	0.0435	0.1739	0.0942	-	-	-
BioASQ_Baseline	0.5000	-	-	-	0.1083	0.0962	0.0951
BioASQ Baseline FS	0.3929	-	-	-	0.1083	0.0962	0.0951

Ideal Answers

	Automatic scores		Manual scores
System	Rouge-2	Rouge-SU4	Readability	Recall	Precision	Repetition
Biomedical Text Ming	0.1846	0.1948	4.26	3.54	4.14	4.69
main system	0.3280	0.3411	3.17	4.05	3.14	3.37
SNUMedinfo2	0.3157	0.3198	3.92	4.15	3.92	4.11
SNUMedinfo1	0.3169	0.3225	3.94	4.12	3.90	4.26
SNUMedinfo3	0.3116	0.3174	3.80	4.01	3.81	4.09
SNUMedinfo4	0.2910	0.2973	3.90	4.05	3.85	4.22
SNUMedinfo5	0.2894	0.2982	3.82	3.91	3.74	3.99
system 2	0.3247	0.3392	1.80	2.27	1.88	1.94
system 3	0.3352	0.3493	1.70	2.05	1.70	1.73
system 4	0.3282	0.3419	1.41	1.71	1.42	1.46
Wishart-S1	0.3914	0.4089	3.41	4.15	3.46	3.45
Wishart-S2	0.3914	0.4089	0.13	0.17	0.13	0.15
system 5	0.3231	0.3385	1.33	1.65	1.35	1.43
BioASQ_Baseline	0.3183	0.3577	-	-	-	-
BioASQ Baseline FS	0.3121	0.3359	-	-	-	-

Test batch 3

Exact Answers

	Yes/No	Factoid			List
System	Accuracy	Strict Acc.	Lenient Acc.	MRR	Mean precision	Recall	F-Measure
Biomedical Text Ming	0.8333	0.0417	0.1250	0.0833	0.1195	0.1780	0.1373
SNUMedinfo1	-	-	-	-	-	-	-
SNUMedinfo2	-	-	-	-	-	-	-
SNUMedinfo3	-	-	-	-	-	-	-
SNUMedinfo4	-	-	-	-	-	-	-
SNUMedinfo5	-	-	-	-	-	-	-
Wishart-S1	0.8889	0.0417	0.0833	0.0556	0.4584	0.3763	0.3909
main system	0.8333	-	-	-	-	-	-
Asclepius	0.8333	0.0417	0.0417	0.0417	-	-	-
BioASQ_Baseline	0.7222	-	-	-	0.1010	0.2340	0.1114
BioASQ Baseline FS	0.6111	-	-	-	0.1010	0.2340	0.1114

Ideal Answers

	Automatic scores		Manual scores
System	Rouge-2	Rouge-SU4	Readability	Recall	Precision	Repetition
Biomedical Text Ming	0.2134	0.2190	4.15	3.30	3.71	4.53
SNUMedinfo1	0.3382	0.3446	4.06	3.94	3.93	4.26
SNUMedinfo2	0.3560	0.3605	3.94	4.01	3.89	4.10
SNUMedinfo3	0.3321	0.3385	3.91	3.85	3.81	4.09
SNUMedinfo4	0.3307	0.3369	3.96	3.82	3.77	4.14
SNUMedinfo5	0.3343	0.3407	3.76	3.83	3.72	4.02
Wishart-S1	0.4331	0.4427	3.23	3.58	3.29	3.25
main system	0.4282	0.4386	3.30	4.04	3.45	3.52
Asclepius	-	-	0.47	0.47	0.47	0.47
BioASQ_Baseline	0.3319	0.3564	-	-	-	-
BioASQ Baseline FS	0.3219	0.3589	-	-	-	-

Test batch 4

Exact Answers

	Yes/No	Factoid			List
System	Accuracy	Strict Acc.	Lenient Acc.	MRR	Mean precision	Recall	F-Measure
Hippocrates	0.8750	0.0625	0.1875	0.1120	-	-	-
Asclepius	0.8750	0.0625	0.2188	0.1354	-	-	-
Biomedical Text Ming	0.8750	0.0938	0.1250	0.1042	-	-	-
SNUMedinfo1	-	-	-	-	-	-	-
SNUMedinfo2	-	-	-	-	-	-	-
SNUMedinfo3	-	-	-	-	-	-	-
SNUMedinfo4	-	-	-	-	-	-	-
SNUMedinfo5	-	-	-	-	-	-	-
Wishart-S1	0.9375	0.2500	0.2813	0.2813	0.2659	0.4029	0.2963
BioASQ_Baseline	0.5000	-	-	-	0.1233	0.1365	0.1062
BioASQ Baseline FS	0.3438	-	-	-	0.1233	0.1365	0.1062

Ideal Answers

	Automatic scores		Manual scores
System	Rouge-2	Rouge-SU4	Readability	Recall	Precision	Repetition
Hippocrates	-	-	-	-	-	-
Asclepius	-	-	0.74	0.74	0.74	0.74
Biomedical Text Ming	0.1796	0.1938	4.01	3.67	3.84	4.45
SNUMedinfo1	0.2963	0.3158	3.90	4.18	4.09	4.14
SNUMedinfo2	0.2858	0.3087	3.82	4.18	4.04	4.10
SNUMedinfo3	0.2927	0.3130	3.84	4.23	4.05	4.15
SNUMedinfo4	0.2756	0.2968	3.79	4.10	3.99	4.12
SNUMedinfo5	0.3164	0.3384	3.86	4.24	4.06	4.04
Wishart-S1	0.4072	0.4295	3.46	4.15	3.88	3.41
BioASQ_Baseline	0.3273	0.3677	-	-	-	-
BioASQ Baseline FS	0.3075	0.3569	-	-	-	-

Test batch 5

Exact Answers

	Yes/No	Factoid			List
System	Accuracy	Strict Acc.	Lenient Acc.	MRR	Mean precision	Recall	F-Measure
SNUMedinfo1	-	-	-	-	-	-	-
SNUMedinfo2	-	-	-	-	-	-	-
SNUMedinfo3	-	-	-	-	-	-	-
SNUMedinfo4	-	-	-	-	-	-	-
SNUMedinfo5	-	-	-	-	-	-	-
Biomedical Text Ming	1.0000	0.1379	0.1724	0.1466	-	-	-
Asclepius	0.8333	0.0345	0.0690	0.0460	-	-	-
BioASQ_Baseline	0.3750	-	-	-	0.1548	0.1665	0.1319
BioASQ Baseline FS	0.4583	-	-	-	0.1548	0.1665	0.1319

Ideal Answers

	Automatic scores		Manual scores
System	Rouge-2	Rouge-SU4	Readability	Recall	Precision	Repetition
SNUMedinfo1	0.2929	0.3178	3.71	3.99	4.09	4.03
SNUMedinfo2	0.2945	0.3185	1.57	1.89	1.82	1.71
SNUMedinfo3	0.2895	0.3146	0.63	0.73	0.76	0.64
SNUMedinfo4	0.2765	0.2984	0.94	0.97	1.06	1.03
SNUMedinfo5	0.2919	0.3163	2.15	2.40	2.42	2.36
Biomedical Text Ming	0.1685	0.1817	3.94	3.70	3.95	4.32
Asclepius	-	-	0.00	0.00	0.00	0.00
BioASQ_Baseline	0.3297	0.3599	-	-	-	-
BioASQ Baseline FS	0.3366	0.3639	-	-	-	-

Copyright © 2023, the BioASQ Challenge (http://bioasq.org)

Platinum sponsors of the BioASQ Challenge between 2012 and 2023

Google was a proud sponsor of the BioASQ Challenge from 2018 to 2022.
BioASQ was supported by the National Library of Medicine of the National Institutes of Health under award number R13LM012214. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.
The research leading to these results has received funding from the European Commission's Seventh Framework Programme (FP7/2007-2013, ICT-2011.4.4(d), Intelligent Information Management, Targeted Competition Framework) under grant agreement n° 318652.