EVALUATION RESULTS
Below are the results of evaluations performed using MHCBench. The data sets used in evaluations are available on this server. The developers may download these data sets and can evaluate their algorithm.
Evaluation with Set1: All Non Redundant Peptides form data bases and literature
Algorithm | EVALUATION PARAMETERS |
---|
Threshold Dependent* | Threshold Independent | For Detailed Results |
Category | Reference | Sens | Spec | PPV | NPV | Accu | CC | ROC |
Motif | Rammennsee et al., 1995 | 69.16 | 71.21 | 83.77 | 51.88 | 69.81 | 0.3790 | 0.707 | [Details] |
Max et al., 1993 | 55.91 | 66.56 | 78.23 | 41.27 | 59.29 | 0.2093 | 0.649 | [Details] |
Sette et al., 1993 | 52.59 | 73.37 | 80.93 | 41.87 | 59.19 | 0.2433 | 0.650 | [Details] |
Hammer et al., 1993 | 64.12 | 58.20 | 76.72 | 43.02 | 62.24 | 0.2100 | 0.638 | [Details] |
Chicz et al., 1993 | 28.82 | 82.66 | 78.12 | 35.09 | 45.92 | 0.1232 | 0.542 | [Details] |
Matrix | Marshal et al., 1995 | 72.19 | 72.14 | 84.77 | 54.69 | 72.17 | 0.4183 | 0.787 | [Details] |
Struniolo et al., 1999 | 73.05 | 67.49 | 82.84 | 53.83 | 71.29 | 0.3856 | 0.777 | [Details] |
Hammer et al., 1994 | 69.45 | 70.59 | 83.54 | 51.82 | 69.81 | 0.3762 | 0.774 | [Details] |
Brusic et al., 1998 | 64.27 | 65.33 | 79.93 | 45.97 | 64.60 | 0.2768 | 0.707 | [Details] |
Southwood et al., 1998 | 66.71 | 66.25 | 80.94 | 48.09 | 66.57 | 0.3094 | 0.700 | [Details] |
Borras-Cuesta et al., 2000 | 61.82 | 60.37 | 77.02 | 42.39 | 61.36 | 0.2075 | 0.664 | [Details] |
ANN | Brusic et al., 1998 | 70.75 | 70.59 | 83.79 | 59.90 | 70.70 | 0.3894 | 0.771 | [Details] |
* The Parameters is calculated at threshold where sensitivity is nearly equal to specificity, only for matrices and ANN.
Evaluation with Set2: Natural Peptides selected form set1
Algorithm | EVALUATION PARAMETERS |
---|
Threshold Dependent* | Threshold Independent | For Detailed Results |
Category | Reference | Sens | Spec | PPV | NPV | Accu | CC | ROC |
Motif | Rammennsee et al., 1995 | 64.30 | 71.92 | 74.92 | 60.69 | 67.61 | 0.3542 | 0.704 | [Details] |
Max et al., 1993 | 50.39 | 67.12 | 66.67 | 50.91 | 57.65 | 0.1755 | 0.619 | [Details] |
Sette et al., 1993 | 51.97 | 70.55 | 69.72 | 52.96 | 60.03 | 0.2260 | 0.630 | [Details] |
Hammer et al., 1993 | 63.25 | 58.22 | 66.39 | 54.84 | 61.07 | 0.2135 | 0.621 | [Details] |
Chicz et al., 1993 | 38.85 | 80.82 | 72.55 | 50.32 | 57.06 | 0.2121 | 0.609 | [Details] |
Matrix | Marshal et al., 1995 | 67.98 | 67.47 | 73.16 | 61.76 | 67.76 | 0.3518 | 0.753 | [Details] |
Struniolo et al., 1999 | 66.93 | 67.12 | 72.55 | 60.87 | 67.01 | 0.3378 | 0.738 | [Details] |
Hammer et al., 1994 | 65.62 | 65.75 | 71.43 | 59.44 | 65.65 | 0.3112 | 0.743 | [Details] |
Brusic et al., 1998 | 64.57 | 64.04 | 70.09 | 58.07 | 64.34 | 0.2838 | 0.695 | [Details] |
Southwood et al., 1998 | 61.42 | 61.99 | 67.83 | 55.18 | 61.66 | 0.2321 | 0.651 | [Details] |
Borras-Cuesta et al., 2000 | 58.27 | 60.27 | 65.68 | 52.54 | 59.14 | 0.1838 | 0.643 | [Details] |
ANN | Brusic et al., 1998 | 65.62 | 65.75 | 71.43 | 49.44 | 65.68 | 0.3112 | 0.685 | [Details] |
* The Parameters is calculated at threshold where sensitivity is nearly equal to specificity, only for matrices and ANN.
Evaluation with Set3a: Non-homologous Peptides selected form set1
Algorithm | EVALUATION PARAMETERS |
---|
Threshold Dependent* | Threshold Independent | For Detailed Results |
Category | Reference | Sens | Spec | PPV | NPV | Accu | CC | ROC |
Motif | Rammennsee et al., 1995 | 60.86 | 74.64 | 84.50 | 52.60 | 65.93 | 0.3428 | 0.681 | [Details] |
Max et al., 1993 | 49.87 | 68.20 | 72.94 | 44.18 | 56.61 | 0.1759 | 0.604 | [Details] |
Sette et al., 1993 | 51.47 | 70.51 | 75.00 | 45.81 | 58.47 | 0.2139 | 0.632 | [Details] |
Hammer et al., 1993 | 62.47 | 58.53 | 72.14 | 47.57 | 61.02 | 0.2034 | 0.605 | [Details] |
Chicz et al., 1993 | 34.85 | 82.95 | 77.84 | 42.55 | 52.54 | 0.1906 | 0.566 | [Details] |
Matrix | Marshal et al., 1995 | 69.17 | 69.12 | 79.38 | 56.60 | 69.15 | 0.3712 | 0.753 | [Details] |
Struniolo et al., 1999 | 69.17 | 68.20 | 78.90 | 58.27 | 68.81 | 0.3626 | 0.734 | [Details] |
Hammer et al., 1994 | 67.29 | 66.36 | 77.47 | 54.14 | 66.95 | 0.3261 | 0.737 | [Details] |
Brusic et al., 1998 | 64.08 | 64.98 | 75.87 | 51.27 | 64.41 | 0.2808 | 0.699 | [Details] |
Southwood et al., 1998 | 64.34 | 63.59 | 75.24 | 50.92 | 64.07 | 0.2703 | 0.677 | [Details] |
Borras-Cuesta et al., 2000 | 62.47 | 63.59 | 74.68 | 49.64 | 62.88 | 0.2518 | 0.685 | [Details] |
ANN | Brusic et al., 1998 | 64.61 | 64.98 | 76.03 | 51.65 | 64.75 | 0.2861 | 0.693 | [Details] |
* The Parameters is calculated at threshold where sensitivity is nearly equal to specificity, only for matrices and ANN.
Evaluation with Set3b: Non-homologous Peptides selected form set2
Algorithm | EVALUATION PARAMETERS |
---|
Threshold Dependent* | Threshold Independent | For Detailed Results |
Category | Reference | Sens | Spec | PPV | NPV | Accu | CC | ROC |
Motif | Rammennsee et al., 1995 | 60.57 | 73.15 | 74.45 | 58.96 | 66.09 | 0.3356 | ----1 | [Details] |
Max et al., 1993 | 50.90 | 68.52 | 67.62 | 51.93 | 58.59 | 0.1948 | 0.628 | [Details] |
Sette et al., 1993 | 53.41 | 70.37 | 69.95 | 53.90 | 60.81 | 0.2381 | 0.655 | [Details] |
Hammer et al., 1993 | 68.87 | 57.41 | 66.30 | 55.86 | 61.62 | 0.2222 | 0.616 | [Details] |
Chicz et al., 1993 | 39.78 | 82.41 | 74.50 | 51.45 | 58.38 | 0.2399 | 0.622 | [Details] |
Matrix | Marshal et al., 1995 | 68.46 | 68.98 | 74.03 | 62.87 | 68.69 | 0.3717 | 0.761 | [Details] |
Struniolo et al., 1999 | 67.03 | 67.59 | 72.76 | 61.34 | 67.27 | 0.3436 | 0.751 | [Details] |
Hammer et al., 1994 | 67.74 | 65.74 | 71.86 | 61.21 | 66.87 | 0.3328 | 0.751 | [Details] |
Brusic et al., 1998 | 63.44 | 64.35 | 69.69 | 57.68 | 63.84 | 0.2758 | 0.711 | [Details] |
Southwood et al., 1998 | 63.44 | 63.43 | 69.14 | 57.32 | 63.43 | 0.2666 | 0.684 | [Details] |
Borras-Cuesta et al., 2000 | 60.57 | 61.57 | 67.06 | 54.73 | 61.01 | 0.2197 | 0.657 | [Details] |
ANN | Brusic et al., 1998 | 65.59 | 65.74 | 71.21 | 59.66 | 65.66 | 0.3110 | 0.713 | [Details] |
* The Parameters is calculated at threshold where sensitivity is nearly equal to specificity, only for matrices and ANN.
Evaluation with Set4a: Randomly selected equal number of HLA-DRB1*0401 binders and non-binders form set1
Algorithm | EVALUATION PARAMETERS |
---|
Threshold Dependent* | Threshold Independent | For Detailed Results |
Category | Reference | Sens | Spec | PPV | NPV | Accu | CC | ROC |
Motif | Rammennsee et al., 1995 | 69.66 | 71.21 | 70.75 | 70.12 | 70.43 | 0.4087 | 0.711 | [Details] |
Max et al., 1993 | 54.18 | 66.56 | 61.84 | 59.23 | 60.37 | 0.2090 | 0.637 | [Details] |
Sette et al., 1993 | 47.37 | 73.37 | 64.02 | 58.23 | 60.37 | 0.2148 | 0.624 | [Details] |
Hammer et al., 1993 | 64.40 | 58.20 | 60.64 | 62.05 | 61.30 | 0.2264 | 0.633 | [Details] |
Chicz et al., 1993 | 28.48 | 82.66 | 62.16 | 53.61 | 55.57 | 0.1326 | 0.535 | [Details] |
Matrix | Marshal et al., 1995 | 71.21 | 71.21 | 71.21 | 71.21 | 71.21 | 0.4241 | 0.778 | [Details] |
Struniolo et al., 1999 | 71.52 | 67.49 | 68.75 | 70.32 | 69.50 | 0.3904 | 0.764 | [Details] |
Hammer et al., 1994 | 66.25 | 68.42 | 67.72 | 66.97 | 67.34 | 0.3468 | 0.760 | [Details] |
Brusic et al., 1998 | 66.25 | 65.94 | 66.05 | 66.15 | 66.10 | 0.3220 | 0.711 | [Details] |
Southwood et al., 1998 | 64.71 | 64.71 | 64.71 | 64.71 | 64.71 | 0.2941 | 0.680 | [Details] |
Borras-Cuesta et al., 2000 | 58.82 | 60.37 | 59.75 | 59.45 | 59.60 | 0.1920 | 0.653 | [Details] |
ANN | Brusic et al., 1998 | 70.28 | 69.97 | 70.06 | 70.19 | 70.12 | 0.4025 | 0.724 | [Details] |
* The Parameters is calculated at threshold where sensitivity is nearly equal to specificity, only for matrices and ANN.
Evaluation with Set4b: Randomly selected equal number of HLA-DRB1*0401 binders and non-binders form set2
Algorithm | EVALUATION PARAMETERS |
---|
Threshold Dependent* | Threshold Independent | For Detailed Results |
Category | Reference | Sens | Spec | PPV | NPV | Accu | CC | ROC |
Motif | Rammennsee et al., 1995 | 68.15 | 71.92 | 70.82 | 69.31 | 70.03 | 0.4010 | ----- | [Details] |
Max et al., 1993 | 51.03 | 67.12 | 60.82 | 57.82 | 59.08 | 0.1839 | 0.619 | [Details] |
Sette et al., 1993 | 52.40 | 70.55 | 64.02 | 59.71 | 61.47 | 0.2333 | 0.633 | [Details] |
Hammer et al., 1993 | 62.33 | 58.22 | 59.87 | 60.71 | 60.27 | 0.2057 | 0.617 | [Details] |
Chicz et al., 1993 | 38.70 | 80.82 | 66.86 | 56.87 | 59.76 | 0.2152 | 0.608 | [Details] |
Matrix | Marshal et al., 1995 | 68.15 | 68.84 | 68.62 | 68.37 | 68.49 | 0.3699 | 0.767 | [Details] |
Struniolo et al., 1999 | 67.47 | 67.81 | 67.70 | 67.58 | 67.64 | 0.3527 | 0.751 | [Details] |
Hammer et al., 1994 | 66.78 | 65.78 | 66.10 | 66.44 | 66.27 | 0.3254 | 0.751 | [Details] |
Brusic et al., 1998 | 63.70 | 64.04 | 63.92 | 63.82 | 63.87 | 0.2774 | 0.692 | [Details] |
Southwood et al., 1998 | 63.36 | 63.70 | 63.57 | 63.48 | 63.50 | 0.2705 | 0.661 | [Details] |
Borras-Cuesta et al., 2000 | 59.59 | 57.53 | 58.39 | 58.74 | 58.56 | 0.1713 | 0.649 | [Details] |
ANN | Brusic et al., 1998 | 64.04 | 64.38 | 64.26 | 64.16 | 64.21 | 0.2842 | 0.672 | [Details] |
* The Parameters is calculated at threshold where sensitivity is nearly equal to specificity, only for matrices and ANN.
Evaluation with Set5a: Recent Ligands selected accroding to set1 criteion
Algorithm | EVALUATION PARAMETERS |
---|
Threshold Dependent* | Threshold Independent | For Detailed Results |
Category | Reference | Sens | Spec | PPV | NPV | Accu | CC | ROC |
Motif | Rammennsee et al., 1995 | 55.71 | 61.70 | 68.42 | 48.33 | 58.12 | 0.1708 | 0.601 | [Details] |
Max et al., 1993 | 47.14 | 63.83 | 66.00 | 44.78 | 53.85 | 0.1087 | 0.607 | [Details] |
Sette et al., 1993 | 21.43 | 78.72 | 60.00 | 40.22 | 44.44 | 0.0018 | 0.551 | [Details] |
Hammer et al., 1993 | 58.57 | 51.06 | 64.06 | 45.28 | 55.56 | 0.0949 | 0.537 | [Details] |
Chicz et al., 1993 | 14.29 | 91.49 | 71.43 | 41.75 | 45.30 | 0.0872 | 0.466 | [Details] |
Matrix | Marshal et al., 1995 | 65.71 | 59.67 | 70.77 | 53.85 | 63.25 | 0.2495 | 0.662 | [Details] |
Struniolo et al., 1999 | 67.14 | 55.32 | 69.12 | 53.06 | 62.39 | 0.2232 | 0.670 | [Details] |
Hammer et al., 1994 | 61.43 | 46.81 | 63.24 | 44.90 | 55.56 | 0.0819 | 0.658 | [Details] |
Brusic et al., 1998 | 60.00 | 61.70 | 70.00 | 50.88 | 60.68 | 0.2129 | 0.603 | [Details] |
Southwood et al., 1998 | 60.00 | 53.19 | 65.62 | 47.17 | 57.26 | 0.1299 | 0.585 | [Details] |
Borras-Cuesta et al., 2000 | 42.86 | 48.94 | 55.56 | 36.51 | 45.30 | 0.0000 | 0.421 | [Details] |
ANN | Brusic et al., 1998 | 61.43 | 72.34 | 76.79 | 55.74 | 65.81 | 0.3314 | 0.695 | [Details] |
* The Parameters is calculated at threshold where sensitivity is nearly equal to specificity, only for matrices and ANN.
Evaluation with Set5b: Recent Ligands selected accroding to set2 criteion
Algorithm | EVALUATION PARAMETERS |
---|
Threshold Dependent* | Threshold Independent | For Detailed Results |
Category | Reference | Sens | Spec | PPV | NPV | Accu | CC | ROC |
Motif | Rammennsee et al., 1995 | 66.67 | 40.54 | 59.26 | 48.39 | 55.29 | 0.0742 | -*--- | [Details] |
Max et al., 1993 | 25.00 | 81.08 | 63.16 | 45.45 | 49.41 | 0.0724 | 0.621 | [Details] |
Sette et al., 1993 | 31.25 | 72.97 | 60.00 | 45.00 | 49.41 | 0.0460 | 0.568 | [Details] |
Hammer et al., 1993 | 41.67 | 64.86 | 60.61 | 46.15 | 51.76 | 0.0664 | 0.512 | [Details] |
Chicz et al., 1993 | 20.83 | 89.19 | 71.43 | 46.48 | 50.59 | 0.1340 | 0.533 | [Details] |
Matrix | Marshal et al., 1995 | 62.50 | 72.97 | 75.00 | 60.50 | 67.06 | 0.3524 | 0.706 | [Details] |
Struniolo et al., 1999 | 60.42 | 59.46 | 65.91 | 53.66 | 60.00 | 0.1972 | 0.677 | [Details] |
Hammer et al., 1994 | 50.00 | 56.76 | 60.00 | 46.67 | 52.94 | 0.0671 | 0.619 | [Details] |
Brusic et al., 1998 | 60.42 | 56.76 | 64.44 | 52.50 | 58.82 | 0.1706 | 0.626 | [Details] |
Southwood et al., 1998 | 50.00 | 59.46 | 61.54 | 47.83 | 54.12 | 0.0941 | 0.691 | [Details] |
Borras-Cuesta et al., 2000 | 56.25 | 40.54 | 55.10 | 41.67 | 49.41 | 0.0000 | 0.439 | [Details] |
ANN | Brusic et al., 1998 | 60.42 | 67.57 | 70.73 | 56.82 | 63.53 | 0.2777 | 0.714 | [Details] |
* The Parameters is calculated at threshold where sensitivity is nearly equal to specificity, only for matrices and ANN.
|