|
EVALUATION RESULTS
Below are the results of evaluations performed using MHCBench. The data sets used in evaluations are available on this server. The developers may download these data sets and can evaluate their algorithm.
Evaluation with Set1: All Non Redundant Peptides form data bases and literature
| Algorithm | EVALUATION PARAMETERS |
|---|
| Threshold Dependent* | Threshold Independent | For Detailed Results |
| Category | Reference | Sens | Spec | PPV | NPV | Accu | CC | ROC |
| Motif | Rammennsee et al., 1995 | 69.16 | 71.21 | 83.77 | 51.88 | 69.81 | 0.3790 | 0.707 | [Details] |
| Max et al., 1993 | 55.91 | 66.56 | 78.23 | 41.27 | 59.29 | 0.2093 | 0.649 | [Details] |
| Sette et al., 1993 | 52.59 | 73.37 | 80.93 | 41.87 | 59.19 | 0.2433 | 0.650 | [Details] |
| Hammer et al., 1993 | 64.12 | 58.20 | 76.72 | 43.02 | 62.24 | 0.2100 | 0.638 | [Details] |
| Chicz et al., 1993 | 28.82 | 82.66 | 78.12 | 35.09 | 45.92 | 0.1232 | 0.542 | [Details] |
| Matrix | Marshal et al., 1995 | 72.19 | 72.14 | 84.77 | 54.69 | 72.17 | 0.4183 | 0.787 | [Details] |
| Struniolo et al., 1999 | 73.05 | 67.49 | 82.84 | 53.83 | 71.29 | 0.3856 | 0.777 | [Details] |
| Hammer et al., 1994 | 69.45 | 70.59 | 83.54 | 51.82 | 69.81 | 0.3762 | 0.774 | [Details] |
| Brusic et al., 1998 | 64.27 | 65.33 | 79.93 | 45.97 | 64.60 | 0.2768 | 0.707 | [Details] |
| Southwood et al., 1998 | 66.71 | 66.25 | 80.94 | 48.09 | 66.57 | 0.3094 | 0.700 | [Details] |
| Borras-Cuesta et al., 2000 | 61.82 | 60.37 | 77.02 | 42.39 | 61.36 | 0.2075 | 0.664 | [Details] |
| ANN | Brusic et al., 1998 | 70.75 | 70.59 | 83.79 | 59.90 | 70.70 | 0.3894 | 0.771 | [Details] |
* The Parameters is calculated at threshold where sensitivity is nearly equal to specificity, only for matrices and ANN.
Evaluation with Set2: Natural Peptides selected form set1
| Algorithm | EVALUATION PARAMETERS |
|---|
| Threshold Dependent* | Threshold Independent | For Detailed Results |
| Category | Reference | Sens | Spec | PPV | NPV | Accu | CC | ROC |
| Motif | Rammennsee et al., 1995 | 64.30 | 71.92 | 74.92 | 60.69 | 67.61 | 0.3542 | 0.704 | [Details] |
| Max et al., 1993 | 50.39 | 67.12 | 66.67 | 50.91 | 57.65 | 0.1755 | 0.619 | [Details] |
| Sette et al., 1993 | 51.97 | 70.55 | 69.72 | 52.96 | 60.03 | 0.2260 | 0.630 | [Details] |
| Hammer et al., 1993 | 63.25 | 58.22 | 66.39 | 54.84 | 61.07 | 0.2135 | 0.621 | [Details] |
| Chicz et al., 1993 | 38.85 | 80.82 | 72.55 | 50.32 | 57.06 | 0.2121 | 0.609 | [Details] |
| Matrix | Marshal et al., 1995 | 67.98 | 67.47 | 73.16 | 61.76 | 67.76 | 0.3518 | 0.753 | [Details] |
| Struniolo et al., 1999 | 66.93 | 67.12 | 72.55 | 60.87 | 67.01 | 0.3378 | 0.738 | [Details] |
| Hammer et al., 1994 | 65.62 | 65.75 | 71.43 | 59.44 | 65.65 | 0.3112 | 0.743 | [Details] |
| Brusic et al., 1998 | 64.57 | 64.04 | 70.09 | 58.07 | 64.34 | 0.2838 | 0.695 | [Details] |
| Southwood et al., 1998 | 61.42 | 61.99 | 67.83 | 55.18 | 61.66 | 0.2321 | 0.651 | [Details] |
| Borras-Cuesta et al., 2000 | 58.27 | 60.27 | 65.68 | 52.54 | 59.14 | 0.1838 | 0.643 | [Details] |
| ANN | Brusic et al., 1998 | 65.62 | 65.75 | 71.43 | 49.44 | 65.68 | 0.3112 | 0.685 | [Details] |
* The Parameters is calculated at threshold where sensitivity is nearly equal to specificity, only for matrices and ANN.
Evaluation with Set3a: Non-homologous Peptides selected form set1
| Algorithm | EVALUATION PARAMETERS |
|---|
| Threshold Dependent* | Threshold Independent | For Detailed Results |
| Category | Reference | Sens | Spec | PPV | NPV | Accu | CC | ROC |
| Motif | Rammennsee et al., 1995 | 60.86 | 74.64 | 84.50 | 52.60 | 65.93 | 0.3428 | 0.681 | [Details] |
| Max et al., 1993 | 49.87 | 68.20 | 72.94 | 44.18 | 56.61 | 0.1759 | 0.604 | [Details] |
| Sette et al., 1993 | 51.47 | 70.51 | 75.00 | 45.81 | 58.47 | 0.2139 | 0.632 | [Details] |
| Hammer et al., 1993 | 62.47 | 58.53 | 72.14 | 47.57 | 61.02 | 0.2034 | 0.605 | [Details] |
| Chicz et al., 1993 | 34.85 | 82.95 | 77.84 | 42.55 | 52.54 | 0.1906 | 0.566 | [Details] |
| Matrix | Marshal et al., 1995 | 69.17 | 69.12 | 79.38 | 56.60 | 69.15 | 0.3712 | 0.753 | [Details] |
| Struniolo et al., 1999 | 69.17 | 68.20 | 78.90 | 58.27 | 68.81 | 0.3626 | 0.734 | [Details] |
| Hammer et al., 1994 | 67.29 | 66.36 | 77.47 | 54.14 | 66.95 | 0.3261 | 0.737 | [Details] |
| Brusic et al., 1998 | 64.08 | 64.98 | 75.87 | 51.27 | 64.41 | 0.2808 | 0.699 | [Details] |
| Southwood et al., 1998 | 64.34 | 63.59 | 75.24 | 50.92 | 64.07 | 0.2703 | 0.677 | [Details] |
| Borras-Cuesta et al., 2000 | 62.47 | 63.59 | 74.68 | 49.64 | 62.88 | 0.2518 | 0.685 | [Details] |
| ANN | Brusic et al., 1998 | 64.61 | 64.98 | 76.03 | 51.65 | 64.75 | 0.2861 | 0.693 | [Details] |
* The Parameters is calculated at threshold where sensitivity is nearly equal to specificity, only for matrices and ANN.
Evaluation with Set3b: Non-homologous Peptides selected form set2
| Algorithm | EVALUATION PARAMETERS |
|---|
| Threshold Dependent* | Threshold Independent | For Detailed Results |
| Category | Reference | Sens | Spec | PPV | NPV | Accu | CC | ROC |
| Motif | Rammennsee et al., 1995 | 60.57 | 73.15 | 74.45 | 58.96 | 66.09 | 0.3356 | ----1 | [Details] |
| Max et al., 1993 | 50.90 | 68.52 | 67.62 | 51.93 | 58.59 | 0.1948 | 0.628 | [Details] |
| Sette et al., 1993 | 53.41 | 70.37 | 69.95 | 53.90 | 60.81 | 0.2381 | 0.655 | [Details] |
| Hammer et al., 1993 | 68.87 | 57.41 | 66.30 | 55.86 | 61.62 | 0.2222 | 0.616 | [Details] |
| Chicz et al., 1993 | 39.78 | 82.41 | 74.50 | 51.45 | 58.38 | 0.2399 | 0.622 | [Details] |
| Matrix | Marshal et al., 1995 | 68.46 | 68.98 | 74.03 | 62.87 | 68.69 | 0.3717 | 0.761 | [Details] |
| Struniolo et al., 1999 | 67.03 | 67.59 | 72.76 | 61.34 | 67.27 | 0.3436 | 0.751 | [Details] |
| Hammer et al., 1994 | 67.74 | 65.74 | 71.86 | 61.21 | 66.87 | 0.3328 | 0.751 | [Details] |
| Brusic et al., 1998 | 63.44 | 64.35 | 69.69 | 57.68 | 63.84 | 0.2758 | 0.711 | [Details] |
| Southwood et al., 1998 | 63.44 | 63.43 | 69.14 | 57.32 | 63.43 | 0.2666 | 0.684 | [Details] |
| Borras-Cuesta et al., 2000 | 60.57 | 61.57 | 67.06 | 54.73 | 61.01 | 0.2197 | 0.657 | [Details] |
| ANN | Brusic et al., 1998 | 65.59 | 65.74 | 71.21 | 59.66 | 65.66 | 0.3110 | 0.713 | [Details] |
* The Parameters is calculated at threshold where sensitivity is nearly equal to specificity, only for matrices and ANN.
Evaluation with Set4a: Randomly selected equal number of HLA-DRB1*0401 binders and non-binders form set1
| Algorithm | EVALUATION PARAMETERS |
|---|
| Threshold Dependent* | Threshold Independent | For Detailed Results |
| Category | Reference | Sens | Spec | PPV | NPV | Accu | CC | ROC |
| Motif | Rammennsee et al., 1995 | 69.66 | 71.21 | 70.75 | 70.12 | 70.43 | 0.4087 | 0.711 | [Details] |
| Max et al., 1993 | 54.18 | 66.56 | 61.84 | 59.23 | 60.37 | 0.2090 | 0.637 | [Details] |
| Sette et al., 1993 | 47.37 | 73.37 | 64.02 | 58.23 | 60.37 | 0.2148 | 0.624 | [Details] |
| Hammer et al., 1993 | 64.40 | 58.20 | 60.64 | 62.05 | 61.30 | 0.2264 | 0.633 | [Details] |
| Chicz et al., 1993 | 28.48 | 82.66 | 62.16 | 53.61 | 55.57 | 0.1326 | 0.535 | [Details] |
| Matrix | Marshal et al., 1995 | 71.21 | 71.21 | 71.21 | 71.21 | 71.21 | 0.4241 | 0.778 | [Details] |
| Struniolo et al., 1999 | 71.52 | 67.49 | 68.75 | 70.32 | 69.50 | 0.3904 | 0.764 | [Details] |
| Hammer et al., 1994 | 66.25 | 68.42 | 67.72 | 66.97 | 67.34 | 0.3468 | 0.760 | [Details] |
| Brusic et al., 1998 | 66.25 | 65.94 | 66.05 | 66.15 | 66.10 | 0.3220 | 0.711 | [Details] |
| Southwood et al., 1998 | 64.71 | 64.71 | 64.71 | 64.71 | 64.71 | 0.2941 | 0.680 | [Details] |
| Borras-Cuesta et al., 2000 | 58.82 | 60.37 | 59.75 | 59.45 | 59.60 | 0.1920 | 0.653 | [Details] |
| ANN | Brusic et al., 1998 | 70.28 | 69.97 | 70.06 | 70.19 | 70.12 | 0.4025 | 0.724 | [Details] |
* The Parameters is calculated at threshold where sensitivity is nearly equal to specificity, only for matrices and ANN.
Evaluation with Set4b: Randomly selected equal number of HLA-DRB1*0401 binders and non-binders form set2
| Algorithm | EVALUATION PARAMETERS |
|---|
| Threshold Dependent* | Threshold Independent | For Detailed Results |
| Category | Reference | Sens | Spec | PPV | NPV | Accu | CC | ROC |
| Motif | Rammennsee et al., 1995 | 68.15 | 71.92 | 70.82 | 69.31 | 70.03 | 0.4010 | ----- | [Details] |
| Max et al., 1993 | 51.03 | 67.12 | 60.82 | 57.82 | 59.08 | 0.1839 | 0.619 | [Details] |
| Sette et al., 1993 | 52.40 | 70.55 | 64.02 | 59.71 | 61.47 | 0.2333 | 0.633 | [Details] |
| Hammer et al., 1993 | 62.33 | 58.22 | 59.87 | 60.71 | 60.27 | 0.2057 | 0.617 | [Details] |
| Chicz et al., 1993 | 38.70 | 80.82 | 66.86 | 56.87 | 59.76 | 0.2152 | 0.608 | [Details] |
| Matrix | Marshal et al., 1995 | 68.15 | 68.84 | 68.62 | 68.37 | 68.49 | 0.3699 | 0.767 | [Details] |
| Struniolo et al., 1999 | 67.47 | 67.81 | 67.70 | 67.58 | 67.64 | 0.3527 | 0.751 | [Details] |
| Hammer et al., 1994 | 66.78 | 65.78 | 66.10 | 66.44 | 66.27 | 0.3254 | 0.751 | [Details] |
| Brusic et al., 1998 | 63.70 | 64.04 | 63.92 | 63.82 | 63.87 | 0.2774 | 0.692 | [Details] |
| Southwood et al., 1998 | 63.36 | 63.70 | 63.57 | 63.48 | 63.50 | 0.2705 | 0.661 | [Details] |
| Borras-Cuesta et al., 2000 | 59.59 | 57.53 | 58.39 | 58.74 | 58.56 | 0.1713 | 0.649 | [Details] |
| ANN | Brusic et al., 1998 | 64.04 | 64.38 | 64.26 | 64.16 | 64.21 | 0.2842 | 0.672 | [Details] |
* The Parameters is calculated at threshold where sensitivity is nearly equal to specificity, only for matrices and ANN.
Evaluation with Set5a: Recent Ligands selected accroding to set1 criteion
| Algorithm | EVALUATION PARAMETERS |
|---|
| Threshold Dependent* | Threshold Independent | For Detailed Results |
| Category | Reference | Sens | Spec | PPV | NPV | Accu | CC | ROC |
| Motif | Rammennsee et al., 1995 | 55.71 | 61.70 | 68.42 | 48.33 | 58.12 | 0.1708 | 0.601 | [Details] |
| Max et al., 1993 | 47.14 | 63.83 | 66.00 | 44.78 | 53.85 | 0.1087 | 0.607 | [Details] |
| Sette et al., 1993 | 21.43 | 78.72 | 60.00 | 40.22 | 44.44 | 0.0018 | 0.551 | [Details] |
| Hammer et al., 1993 | 58.57 | 51.06 | 64.06 | 45.28 | 55.56 | 0.0949 | 0.537 | [Details] |
| Chicz et al., 1993 | 14.29 | 91.49 | 71.43 | 41.75 | 45.30 | 0.0872 | 0.466 | [Details] |
| Matrix | Marshal et al., 1995 | 65.71 | 59.67 | 70.77 | 53.85 | 63.25 | 0.2495 | 0.662 | [Details] |
| Struniolo et al., 1999 | 67.14 | 55.32 | 69.12 | 53.06 | 62.39 | 0.2232 | 0.670 | [Details] |
| Hammer et al., 1994 | 61.43 | 46.81 | 63.24 | 44.90 | 55.56 | 0.0819 | 0.658 | [Details] |
| Brusic et al., 1998 | 60.00 | 61.70 | 70.00 | 50.88 | 60.68 | 0.2129 | 0.603 | [Details] |
| Southwood et al., 1998 | 60.00 | 53.19 | 65.62 | 47.17 | 57.26 | 0.1299 | 0.585 | [Details] |
| Borras-Cuesta et al., 2000 | 42.86 | 48.94 | 55.56 | 36.51 | 45.30 | 0.0000 | 0.421 | [Details] |
| ANN | Brusic et al., 1998 | 61.43 | 72.34 | 76.79 | 55.74 | 65.81 | 0.3314 | 0.695 | [Details] |
* The Parameters is calculated at threshold where sensitivity is nearly equal to specificity, only for matrices and ANN.
Evaluation with Set5b: Recent Ligands selected accroding to set2 criteion
| Algorithm | EVALUATION PARAMETERS |
|---|
| Threshold Dependent* | Threshold Independent | For Detailed Results |
| Category | Reference | Sens | Spec | PPV | NPV | Accu | CC | ROC |
| Motif | Rammennsee et al., 1995 | 66.67 | 40.54 | 59.26 | 48.39 | 55.29 | 0.0742 | -*--- | [Details] |
| Max et al., 1993 | 25.00 | 81.08 | 63.16 | 45.45 | 49.41 | 0.0724 | 0.621 | [Details] |
| Sette et al., 1993 | 31.25 | 72.97 | 60.00 | 45.00 | 49.41 | 0.0460 | 0.568 | [Details] |
| Hammer et al., 1993 | 41.67 | 64.86 | 60.61 | 46.15 | 51.76 | 0.0664 | 0.512 | [Details] |
| Chicz et al., 1993 | 20.83 | 89.19 | 71.43 | 46.48 | 50.59 | 0.1340 | 0.533 | [Details] |
| Matrix | Marshal et al., 1995 | 62.50 | 72.97 | 75.00 | 60.50 | 67.06 | 0.3524 | 0.706 | [Details] |
| Struniolo et al., 1999 | 60.42 | 59.46 | 65.91 | 53.66 | 60.00 | 0.1972 | 0.677 | [Details] |
| Hammer et al., 1994 | 50.00 | 56.76 | 60.00 | 46.67 | 52.94 | 0.0671 | 0.619 | [Details] |
| Brusic et al., 1998 | 60.42 | 56.76 | 64.44 | 52.50 | 58.82 | 0.1706 | 0.626 | [Details] |
| Southwood et al., 1998 | 50.00 | 59.46 | 61.54 | 47.83 | 54.12 | 0.0941 | 0.691 | [Details] |
| Borras-Cuesta et al., 2000 | 56.25 | 40.54 | 55.10 | 41.67 | 49.41 | 0.0000 | 0.439 | [Details] |
| ANN | Brusic et al., 1998 | 60.42 | 67.57 | 70.73 | 56.82 | 63.53 | 0.2777 | 0.714 | [Details] |
* The Parameters is calculated at threshold where sensitivity is nearly equal to specificity, only for matrices and ANN.
|