youden index roc curveyouden index roc curve

Epub 2017 Dec 15. Markers no longer require SAS 9.4 TS1M1 or later. r - Ho to make the Receiver operating characteristic (ROC) and J Biopharma Stat. (5) and the convergences in eqs. I got this example below and wondering how to get the optimal threshold (Youden's index = sensitivity+specificity-1) for each method and plot that value on the ROC curve to know the coordinate obtained from that optimal threshold. Stat. Description In this graph (part of ROC curve analysis) you can plot the following statistics against the criterion values: Sensitivity and specificity, and optionally their 95% Confidence Intervals Youden index The Youden index for a single point on the ROC curve is defined as sensitivity + specificity - 1 Another thinning method is available that can be used in addition to the above method (or instead of, by specifying THINSENS=1 and THINY=1). It was recently pointed out that the SROC curve is in principle unidentifiable if only one (Sp, Se) pair per study is known. (2000), "Principles and practical application of the receiver-operating characteristic analysis for diagnostic tests," Preventive Veterinary Medicine, 45, 23-41. (5) and (9), is easy to derive that. 1997;53:82337. Martnez-Camblor P, Pardo-Fernndez J. In biomarker development, levels may be unquantifiable below a limit of detection (LOD) and missing from the overall dataset. Theoretically can the Ackermann function be optimized? 2013;12:1727. Greiner, M., Pfeiffer D., Smith R.D. Bivariate random effects meta-analysis of ROC curves. The receiver operating characteristic (ROC) curve and their associated summary indices, such as the Youden index, are statistical tools commonly used to analyze the discrimination ability of a (bio)marker to distinguish between two populations. If it is unable to do this (such as if there is no active internet connection available), the macro will issue the following message: The computations performed by the macro are not affected by the appearance of this message. 2004 Sep;57(9):925-32. doi: 10.1016/j.jclinepi.2003.12.019. This paper presents the concept of Youden index in the context of the generalized ROC (gROC) curve for non-monotone relationships. Before While the predicted probability is a continuous value between 0 and 1, it is often desirable to provide a binary prediction of whether the event will occur or not. Pharma Stat. Zweig M. H., Campbell G. Receiver-operating characteristic (ROC) plots: a . r - Identifying threshold from Youden Index - Using ROC curve to Greiner et. Added FORMAT= option specifying the format for numeric values in labels. Youden.index The maximum Youden index value. 2023 Jun 14;18(6):e0286982. This point is identified on the ROC plot by the symbol, D. This symbol is colored red and made larger and bolder by the OPTSYMBOLSTYLE= option. youden: Computes optimal cutoff point using the Youden index criteria Index for rating diagnostic tests. The THINSENS=0 option is removed to reduce crowding of labels. To summarize an ROC curve and determine its optimal cut-point, the Youden index is popularly used. Replaced LINECOLOR=, LINEPATTERN=, and LINETHICKNESS= options with LINESTYLE= option. By including _OPTCORR_ in the ID= option, the cutpoint with maximum correct classification rate is also identified in the ROC plot by adding the symbol, C, in that cutpoint's label. The Youden Index in the Generalized Receiver Operating Characteristic Curve Context. In this paper we compare several estimation pr How does "safely" function in this sentence? Comput Stat Data Anal. . Our approach leads in a natural way to a new, model-based proposal for a summary ROC curve. Do axioms of the physical and mental need to be consistent? The disadvantage is that there could be multiple cutpoints for a given optimality criterion, and as stated, only observed values of the predictor can be selected as optimal. J Clin Epidemiol. Added MARKERS= option to turn point markers on or off at labeled points (available in SAS 9.4 TS1M1 or later). The .gov means its official. This call of the ROCPLOT macro produces an ROC plot of the fitted model using the default appearance. Careers. On smoothing and the bootstrap. Youden Index and the optimal threshold for markers with mass at zero Not the answer you're looking for? The receiver operating characteristic (ROC) curve and their associated summary indices, such as the Youden index, are statistical tools commonly used to analyze the discrimination ability of a (bio)marker to distinguish between two populations. J Clin Epidemiol. Let Y0 and Y1 denote the biomarker in the negative population and in the positive population, respectively. It is recommended that you name the file rocplot.sas. Then, direct calculations lead to deduce that the interval leading to the the value of 1specificity for Jg is achieved is of the form, These points are the ones where the normal densities associated to the positive and negative populations cross each other. 2019 Walter de Gruyter GmbH, Berlin/Boston. The cutpoint at Age 65 also minimizes the difference between the sensitivity and specificity (0.148). r(tY)24(1+)1/3n1/3{tYtY}LnargmaxzR{Z(n)(z)z2}. 29:294655. THINY=-1 turns off this requirement. Asking for help, clarification, or responding to other answers. Epub 2018 Oct 25. These statements fit the logistic model with three predictors. Similarly for the efficiency criterion which has an optimal cutpoint at each prevalence, these cutpoints are presented in the "Efficiency-Based Optimal Cutpoints" table. The code requires SPSS v 12 or above. Contrast Media Mol Imaging. Additionally for each of the cost, MCT and efficiency criteria, all of the optimal cutpoints for the specified cost ratio and/or prevalences can be plotted by specifying the MULTOPTPLOT=YES or MULTOPTPLOT=PANELALL option. Med Decis Making 1999;19:7889.10.1177/0272989X9901900110Search in Google Scholar "The Youden Index in the Generalized Receiver Operating Characteristic Curve Context". Context 1 . By assuming a distribution for the predictor (typically, a normal distribution), the ROC curve is then continuous and monotonic. Since the points on the ROC curve are the distinct predicted probabilities for the observed data, there are points for each combination of A and B levels, such as A1B1, A1B2, A2B1, and A2B2. In this example, data on plants were gathered from four blocks. Notes on the overlap measure as an alternative to the Youden index: How Renamed OUT= option to INPRED=, OUTROC= option to INROC=, PLOTCHAR= option to MARKERSYMBOL=. The data for the following appears in the example titled "Generalized Additive Model with Binary Data" in the PROC GAM documentation. The OPTBYX=PANELALL option requests a single panel of the four plots. When the cost ratio equals 1 and the prevalence equals 0.5, the cost criterion reduces to the Youden index. Martnez-Camblor P, Corral N, Rey C, Pascual J, Cernuda-Morolln E. Receiver operating characteristic curve generalization for non-monotone relationships. If two points have specificity values that differ by less than the MINDIST= value, one of the points is not labeled. The message should occur only if cutpoints do not match in the OUT= and OUTROC= data sets, presumably because the ROCEPS= was not used as suggested in the message. Biom J. PDF Generating Receiver Operating Characteristic (ROC) curve - PharmaSUG To increase the number of points that are labeled by default: Some experimentation with these options might be needed to achieve the desired labeling pattern. To learn more, see our tips on writing great answers. J Korean Stat Soc. When the same cutpoint optimizes multiple criteria, multiple symbols are overlaid and are hard to distinguish. The Youden Index in the Generalized Receiver Operating Characteristic Curve Context, Downloaded on 28.6.2023 from https://www.degruyter.com/document/doi/10.1515/ijb-2018-0060/html, Classical and Ancient Near Eastern Studies, Library and Information Science, Book Studies, The International Journal of Biostatistics, generalized receiver-operating characteristic (gROC) curve, receiver-operating characteristic (ROC) curve, 10.1002/1097-0142(1950)3:1<32::AID-CNCR2820030106>3.0.CO;2-3, Estimating the Population Average Treatment Effect in Observational Studies with Choice-Based Sampling, Significance Tests for Boosted Location and Scale Models with Linear Base-Learners, Hazard Ratio Estimators after Terminating Observation within Matched Pairs in Sibling and Propensity Score Matched Designs, Parametric Regression Analysis with Covariate Misclassification in Main Study/Validation Study Designs, The Performance of Fixed-Horizon, Look-Ahead Procedures Compared to Backward Induction in Bayesian Adaptive-Randomization Decision-Theoretic Clinical Trial Design, Variational Inference for Coupled Hidden Markov Models Applied to the Joint Detection of Copy Number Variations, A Joint Poisson State-Space Modelling Approach to Analysis of Binomial Series with Random Cluster Sizes, Semiparametric Inference for Proportional Mean Past Life Model. It is also possible for two or more points on the ROC curve to have the same label. The MULTOPTLIST= option requests tables of the optima at each cost and/or prevalence for these criteria. Unauthorized use of these marks is strictly prohibited. The interval estimation of the Youden index and the associated cutoff points in a parametric (binormal) and a non-parametric setting is considered. Calculate performance measures at the Youden Index for a ROC curve Description. The Youden index is the value that optimizes sensitivity and specificity for the differentiation. 2013;28:87592.10.1007/s00180-012-0333-1Search in Google Scholar, [17] Spanos A, Harrell FE, Durack DT. The Receiver Operating Characteristic (ROC) curve is a key tool for diagnostic test and has been used in identification of early clinical responses that could predict long-term outcomes. The ROC plots and analyses available in PROC LOGISTIC and the ROCPLOT macro use the empirical ROC curve. Let i and Si be the sample mean and the sample standard deviation, which estimate i and i, respectively (i {0,1}). sens The sensitivity corresponding to the optimal cutoff value. New York: Wiley Blackwell, 2002.10.1002/9780470317082Search in Google Scholar, [2] Hanley JA, McNeil BJ. PMC The sample sizes satisfy. If the predicted probability exceeds the chosen cutpoint, then the event response is predicted. By default, a symbol is placed on the ROC curve for each criterion specified in the OPTCRIT= option. Stat Sin. 2018; In press. Therefore, the Youden index is. When both requirements are turned off, the ROCPLOT macro typically labels the same cutpoints as PROC LOGISTIC when labeling is requested. observations from Y0 and n i.i.d. Arch Rehabil Res Clin Transl. Coffin M, Sukhatme S. Receiver operating characteristic studies and measurement errors. The receiver operating characteristic (ROC) curve is used to evaluate a biomarker's ability for classifying disease status. How to do that? The following are optional parameters except as noted: Parameters for displaying optimal cutpoints. MR-1912205. MeSH For plotting the ROC curve with labeled points, the ROCPLOT macro provides these capabilities: The version of the ROCPLOT macro that you are using is displayed when you specify version (or any string) as the first argument. Youden Index - an overview | ScienceDirect Topics The cutpoint label text is made larger and colored dark blue by the LABELSTYLE= option. R J. PLoS One. 29:294655.10.1002/sim.4044Search in Google Scholar Do physical assets created directly from GPLed, copyleft digital designs (not programs or libraries) acquire the same license? _SESPDIFF_, the absolute difference between sensitivity and specificity. Youden Index Calculator - MDApp - Your Source Of Medical Algorithms Specifying OPTCRIT=CORRECT requests the correct classification criterion, and adding _OPTCORR_ in the ID= option adds "C" to the label of the cutpoint with the maximum correct classification rate. PubMed Central, [14] Martnez-Camblor P, Prez-Fernndez S, Daz-Coto S. Improving the biomarker diagnostic capacity via functional transformations. Any given cutpoint can correspond to many observations in the data. The authors thank too anonymous reviewers for their constructive comments and suggestions. 2013;42:7181. Similarly, the default FORMAT=BEST5. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Duplicate labels and using statistics in labels. observations from Y1 are available. First, a cutpoint adjacent to a labeled cutpoint must have a minimum change in sensitivity (vertical separation) before it is labeled. Tei Index Complements Conventional Echocardiographic Parameters in Diagnostic Workup of Suspected Takotsubo Syndrome. Similarly, if the model involves multiple predictors, then even if an observation's cutpoint is uniquely optimal under a given criterion, it is a function of more than just a single predictor so a confidence interval around one of the predictors is not helpful. Specifying SIZE=0 in the OPTSYMBOLSTYLE= option effectively turns off the symbols on the ROC curve. Martnez-Camblor, P. and Pardo-Fernndez, J. PDF 231-2008: A Macro for Getting More out of Your ROC Curve - SAS Support Hamui RM, Aquino EML, Menezes GMS, Velho Barreto de Arajo T, Seabra Soares de Britto E Alves MT, Valongueiro Alves S, Almeida MDCC. Birkhuser Verlag, Basel, second edition, 2002. Right-click on the link below and select Save to save the ROCPLOT macro definition to a file. Martnez-Camblor P, Pardo-Fernndez JC.. Parametric estimates for the receiver-operating characteristic curve generalization for non-monotone relationships. Try using Medcalc software, it shows the sensitivity, specificity, and the cut-off for Youden index ROC curve analysis. Label of bottom point is below-right. Assume that two independent samples of m of i.i.d. When the model involves a single predictor and the cutpoint that is optimal under a given criterion is unique among the observations in your data, then the following shows how to obtain a confidence interval for the optimal cutpoint on the scale of the predictor. Annal Stat. Note that the cutpoints at Age 65 and 75 (0.716 and 0.952) give the highest correct classification rate (87%). Mossman D. Three-way ROCs. This work is financially supported by the Grants MTM2014-55966-P and MTM2017-89422-P Spanish Ministry of Economy, Industry and Competitiveness; State Research Agency; and FEDER funds. If the points are labeled using only the value of A, then the distinct points A1B1 and A1B2 are both labeled 1. By clicking Post Your Answer, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct. Such explanatory factors (covariates) may influence the ROC curve, the Youden Index . Cancer 1950;3:325.10.1002/1097-0142(1950)3:1<32::AID-CNCR2820030106>3.0.CO;2-3Search in Google Scholar, [7] Coffin M, Sukhatme S. Receiver operating characteristic studies and measurement errors. Pharma Stat. Some criteria do not depend on the cost ratio or the prevalence. Epub 2022 Dec 26. 1996;6:4762.Search in Google Scholar, [25] DasGupta A. Asymptotic theory of statistics and probability. Summary ROC curve based on a weighted Youden index for selecting an Differential diagnosis of acute meningitis: an analysis of the predictive value of initial observations. The "Efficiency-Based Optimal Cutpoints" table shows that the efficiency is maximized at the same EC values as cost and MCT. THINSENS=0 turns off this requirement. Adding this option labels the CELL=0.9 cutpoint whose height above the diagonal does not exceed the default THINY=0.5 requirement. Annal Stat. BMC Infect Dis. Specify the THINVAR= and MINDIST= options. For simplicity, I use the example below instead. 2012 May 1; 56(5): 1103-1114. This call to the ROCPLOT macro plots the ROC curve and labels the points using the predicted probabilities (the cutpoint values) and Age values. (6) and. Diagnostic Accuracy of Fecal Calprotectin for Predicting Relapse in Inflammatory Bowel Disease: A Meta-Analysis. On the other hand, if tY is the point which maximizes (Rg(t)t), then from eqs. Martnez-Camblor, P. & Pardo-Fernndez, J. Bookshelf The results show that the cutpoint at ENTRY=10, LAT=7, LNG=8 maximizes the correct classification rate (0.658) and also minimizes the distance to the "perfect" point (0.531). Is there an extra virgin olive brand produced in Spain, called "Clorlina"? Prediction of malignant lymph nodes in NSCLC by machine-learning classifiers using EBUS-TBNA and PET/CT. For example: The ROCPLOT macro always attempts to check for a later version of itself. Measures of Diagnostic Accuracy: Basic Definitions - PMC I want to get the optimal threshold from ROC curve using Python. References. 2005 Jul 15;24(13):2025-40. doi: 10.1002/sim.2103. The figure I showed is my own ROC curve, but the example I showed is a simple example to make it easy to reproduce. 1989;17:692704.10.1214/aos/1176347135Search in Google Scholar, [23] Schisterman EF, Perkins N. Confidence intervals for the Youden index and corresponding optimal cut-point. PubMed 8600 Rockville Pike This site needs JavaScript to work properly. Consequently, the optimal cutpoint on any criterion found the ROCPLOT macro is restricted to one of the observations. We evaluate an alternative approach, modeling the study-specific ROC curves based on the assumption of linearity in logit space. The OUTPUT statement is added to save the predicted probabilities from the model. The following statements fit the binary logistic model and save the predicted probabilities (variable PHAT) in data set OUT and the ROC information in data set ROC1. Roc curve and cut off point. Observations with a given value of A have different predicted probabilities depending on their values for B. Since CELL is the only predictor in the model, there is one predicted probability and therefore one cutpoint for each value of CELL. DasGupta A. Asymptotic theory of statistics and probability. Then, we directly have that JgP=Jg(1,S1,0,S0). 2012;31:267686.10.1002/sim.4509Search in Google Scholar, [9] Lpez-Ratn M, Cadarso-Surez C, Molanes-Lpez EM, Letn E. Confidence intervals for the symmetry point: an optimal cutpoint in continuous diagnostic tests. The meaning and use of the area under a receiver operating characteristic (ROC) curve. PURPOSE: Produce a plot of the Receiver Operating Characteristic (ROC) curve associated with a fitted binary-response model. 2007;36:54963.10.1080/03610910701212181Search in Google Scholar, [24] Hsieh F, Turnbull BW. Be careful to reproduce the original model. Stat Methods Med Res. However, it is still a matter of debate how to define a summary ROC (SROC) curve. If the original model is a logistic model, specify the DIST=LOGISTIC option in the MODEL statement. A setting is considered in which the pair (Sp, Se) that is selected for publication in a particular study maximizes a weighted Youden index Se+(1-)Sp with a given weight .This leads to a fixed slope (1-)/ of the ROC curve in (1-Sp, Se), equivalent to a slope of (1-)Sp(1-Sp)/(Se(1-Se)) for the corresponding straight line in logit space. Added ALTAXISLABEL= option to use true and false positive rates in axis labels. Stat Med. The ROCPLOT macro enables you to thin the labels using the THINSENS=, THINY=, and the combination of the THINVAR= and MINDIST= options. Stat Meth Med Res. I know how to do it in R with the coords function but I can't seem to find a similar one in Python. PubMed, [21] Shan G. Improved confidence intervals for the Youden index. The Youden Index (J), the maximum potential effectiveness of a biomarker, is a common summary measure of the ROC curve. Defining an Optimal Cut-Point Value in ROC Analysis: An Alternative Troubleshooting Problem Can SPSS compute Youden's index, which can be expressed as sensitivity + specificity -1? J Korean Stat Soc. Bivariate analysis of sensitivity and specificity produces informative summary measures in diagnostic reviews. Now displays markers only at labeled points by default. Was it widely known during his reign that Kaiser Wilhelm II had a deformed arm? But there can be one or more observations at any given value of CELL as can be seen in these results from PROC MEANS. For the other criteria, the single cutpoint (or multiple if tied) that optimizes the criterion also appears in this table. Apparently, the package "OptimalCutpoints" should be able to do it. Your purchase has been completed. Find centralized, trusted content and collaborate around the technologies you use most. Notice that the default CHARLEN=5 option truncates the character values in the labels to the first five characters. Options from previous version are converted as best as possible to new syntax. Receiver operating characteristic curve with the Youden index (J The "Optimal Cutpoints" table shows that the optimal cutpoint occurs at EC=0.364, with sensitivity and specificity values of 0.96 and 0.947 for a difference of 0.013. It is needed only once in a SAS session. The number of cutpoints labeled by this method is given in a note displayed in the SAS log. official website and that any information you provide is encrypted Label of top point is above-left. See the Results tab for examples. Assume that the resulting gROC curve, Rg, satisfies. our work, we are interested in two per- formance indices that can be deducted from the ROC curve: Points on curve closest to the (0,1) and Youden Index ( J ) (see Figure 2. Plot the Receiver Operating Characteristic (ROC) curve associated with a fitted binary response model, reduce (thin) the number of point labels, and find optimal cutpoints using several optimality criteria. Added special variables for sensitivity, specificity, and observation number for use as ID= variables. Liu X. Disclaimer. Geometry nodes - Material Existing boolean value. 20180060. This is the distance from the "perfect" point at the upper-left corner of the ROC plot where 1-Specificity=0 and Sensitivity=1, . This paper presents the concept of Youden index in the government site. 1996;6:4762. Zhou H, Qin G. Confidence intervals for the difference in paired Youden indices. The distance to the "perfect" point in the upper-left corner of the plot is minimum (0.067) at EC=0.364 The "MCT-Based Optimal Cutpoints" and "Cost-Based Optimal Cutpoints" tables show that the cost and MCT criteria are minimized at EC=0.49 when prevalence=0.25 and at EC=0.702 when prevalence=0.05 irrespective of cost ratio. In addition, we have the following convergences in distribution: Besides, the independence among the four random variables is also well-known [25]. Martnez-Camblor P, de Ua lvarez J. An ROC analysis is done to assess the diagnostic and find an optimal cutpoint for determining when the infection is present. and transmitted securely. The interval estimation of the Youden index and the associated cutoff points in a parametric (binormal) and a non-parametric setting is considered. The Youden Index in the Generalized Receiver Operating Characteristic Curve Context. Shan G. Improved confidence intervals for the Youden index. Then. ROC curve, on curve closest to the (0,1) and Youden Index Revista Colombiana de Estadstica 2011;34:13346.Search in Google Scholar, [11] Khanafer N, Sicot N, Vanhems P, Dumitrescu O, Meyssonier V, Tristan A, Bes M, Lina G, Vandenesch F, Gillet Y, Etienne J. eCollection 2023. Added OUTROCDATA= option to name the final data used for graphing. analemma for a specified lat/long at a specific time of day? (Bio)markers; Youden index; cutoff point; generalized receiver-operating characteristic (gROC) curve; receiver-operating characteristic (ROC) curve. Specify the variable to plot the criteria against in the X= option. For a given cutpoint, the first and last values found can depend on the specified order of the character variables in the ID= option. Bethesda, MD 20894, Web Policies Delays in obtaining hospital care and abortion-related complications within a context of illegality. Taking into account basic properties of the Brownian bridge and A1, it can easily be shown that the random variable eq. Please login or register with De Gruyter to order this product. Stat Meth Med Res 2017 DOI: 10.1177/0962280217747009Search in Google Scholar, [6] Youden WJ. A diagnostic test (EC) was used as a predictor of infection. Data set TRAIN contains the data from the first three blocks and is used as the training data set. Second, a cutpoint must have a Youden value greater than a specified proportion of the maximum observed Youden value. Stat Meth Med Res. By including _OPTCORR_, _OPTDIST_, _OPTY_, and _OPTSESP_ in the ID= option, a symbol is added to the label of the cutpoints that optimize the correct classification rate, the distance to the (0,1) point, the Youden index, and the absolute difference between sensitivity and specificity. 2018.10.32614/RJ-2018-043Search in Google Scholar, [16] Martnez-Camblor P, de Ua lvarez J. PROC LOGISTIC can label the points in the ROC curve with their observation numbers, predicted probabilities, values from one or more variables, or with values from a single statistic (such as the misclassification rate). Use the PLOTS=ROC(ID=keyword | ID) option and ID statement. Note that requested optimal cutpoints are labeled even when maximal thinning turns off all cutpoint labeling. The advantage to this is that no distribution assumptions are made in the ROC analysis and curve construction. Accessibility I got this example below and wondering how to get the optimal threshold (Youden's index = sensitivity+specificity-1) for each method and plot that value on the ROC curve to know the coordinate obtained from that optimal threshold. Lpez-Ratn M, Cadarso-Surez C, Molanes-Lpez EM, Letn E. Confidence intervals for the symmetry point: an optimal cutpoint in continuous diagnostic tests. The following references make use of distributional assumptions to produce the ROC curve or estimate an optimal cutpoint and confidence interval. Are there any MTG cards which test for first strike? The Youden Index is a frequently used summary measure of the ROC (Receiver Operating Characteristic) curve. New York: Wiley Blackwell, 2002. 2007;36:54963. Then, we have the following weak convergences. Multi-omics profiling of papillary thyroid microcarcinoma reveals different somatic mutations and a unique transcriptomic signature. Epub 2008 Jun 30. Can SPSS produce Youden's index? - IBM Plot optimality criteria against a selected variable. (1978), "Basic principles of ROC analysis," Seminars in Nuclear Medicine, 8(4), 283-298. The PROC LOGISTIC statements that follow use the OUTROC= option to save ROC information from the model specified in the MODEL statement (in this case, an intercept-only model) and from the generalized additive model. Nonparametric methods for evaluating diagnostic tests. Youden's index is often used in conjunction with receiver operating characteristic (ROC) analysis. 2022 Jul 3;4(3):100213. doi: 10.1016/j.arrct.2022.100213. One of the commonly used method is the Youden index (J) method . Short story in which a scout on a colony ship learns there are no habitable worlds. Revista Colombiana de Estadstica 2011;34:13346. R J. One of the commonly used method is the Youden index () method [ 5 ]. The labels also show the mean SMEAR and maximum BLAST values at each cutpoint matching the results seen in the PROC MEANS results above. For example, adding _OPTY_ in the ID= option will ensure that the cutpoint(s) with maximum Youden index are labeled and that their labels include the symbol, Y. Probability and its Applications. 2018. 25018 - Plot ROC curve with cutpoint labeling and optimal cutpoint analysis This example appears in the LOGISTIC documentation.