Model Assessment¶
The Model Assessment utility enables users to compare continuous predictive models based on the standard version of JMP software. Previously, only the JMP Pro version had supported the ability to compare models (using the Analyze > Predictive Modeling > Model Assessment feature).
The approach to model comparison using the utility in JMP standard is different than in JMP Pro, but the objective is for both versions to yield useful results. Even users who typically use JMP Pro may find the utility based on JMP standard to be beneficial in some scenarios.
To compare predictive models in a data table using the Model Assessment utility:
-
In JMP standard, open the Predictum menu > Self-Validating Ensemble Modeling > Utilities > Model Assessment.
The Model Assessment dialog appears.
-
Select prediction model columns of interest and enter them as Prediction Formulas.
Note: The prediction columns must be continuous.
-
Click OK.
A Model Assessment report appears. For example:
The graph at the top shows the Training RASE (root average squared error) values of the prediction formulas in comparison to the slope of a Passing-Bablok fit on their corresponding Actual by Predicted plots. If all the prediction formulas come from SVEM runs that used the Leave K Out methodology, Average OOB RASE (out-of-bag root average squared error) is used instead of Training RASE. Hovering over a point on the graph will show a tooltip summary of details about the point and the associated Actual by Predicted plot. Clicking on the Actual by Predicted plot in the tooltip will open a separate window, which displays the plot in greater detail.
The Model Performance Summary table displays training RASE values for each prediction equation, and Intercept and Slope values from the Actual by Predicted plots in a table in the next section of the report. If the prediction formulas are from SVEM runs, the table will include Average OOB RASE and PredSE Range. Smaller values are desirable for Training RASE, Average OOB RASE, and PredSE Range. Intercept values near 0 and slopes near 1 are desirable.
The Actual by Predicted plots section is collapsed by default. Expand the section to see the plots. The plots show how well the actual and predicted values correspond. For a perfect fit, all points fall on the diagonal line. The fits are performed with Passing-Bablok method in JMP version 17 or higher, and fit orthogonal with equal variances for earlier versions of JMP. These are the same plots featured in tooltips when hovering over points of the RASE vs. Slope graph.
