Neural Networks Model Pathway

SVEM models that are defined as ensembles of Neural net fits can be constructed in both JMP Standard and JMP Pro. The Neural pathway enables you to fit flexible (non-linear) models.

Launching SVEM

To fit an ensemble model using neural network fits:

1. Open the Predictum menu > Self-Validating Ensemble Modeling > Neural.

   

2. If you don't have a JMP data table open, SVEM prompts you to open one from a local directory.

   The Self-Validating Ensemble Modeling - Neural window appears.

SVEM Neural Launch Window

To define the responses and predictors:

1. Select one or more columns from the Select Columns list and click Y, Response.

2. Select one or more columns from the Select Columns list and click X, Factor.

   

   If you are running JMP Pro, you have the option of selecting the Informative Missing checkbox. For more information about the Informative Missing feature, see the JMP documentation for Informative Missing in the Neural Launch Window section.

   If you are running JMP standard, the Informative Missing checkbox is not featured in the launch dialog. This is because the Neural platform in JMP standard does not support the Informative Missing option.

3. Optionally, select a numeric column for the frequency role from the Select Columns list and click Freq. For each row, the value in a Freq column reflects how many times that row should be included in the analysis. Fractional frequencies are allowed. Frequencies can be used to down-weight specific rows in a SVEM analysis. For example, if you have observations taken under uncontrolled conditions and you want to include them in a model along with observations taken under controlled conditions, you might want to assign all controlled observations a weight of one and the uncontrolled observations a weight less than one.

4. Click Run.

   The Model Launch window appears.

Model Launch Window

You can specify fitting options for the bootstrapped models as well as SVEM settings in this window. These fitting options will apply to all responses.

Model Launch window in JMP standard	Model Launch window in JMP Pro

Note

The SVEM model launch dialog contains options that are available in the Neural platform of JMP. For information about these options, see the JMP documentation for Model Launch Control Panel in the Launch the Neural Platform section.

Changing the Settings

You can change SVEM's settings in the SVEM settings panel box.

Number of Boostrapped Models

In the Number of Bootstrapped Models field, enter the number of bootstrapped models that will be included in the ensemble model.

The default value is 100 for continuous responses or 50 for categorical responses, which proves adequate in many cases. However, you can increase this number if you want greater precision. Note that increasing the number of bootstrapped models also increases the runtime.

Set Random Seed

To obtain a reproducible analysis, enter an integer value in the Set Random Seed field. You can use this seed to rerun the analysis and obtain the same results. The seed is given in the Model Summary outline box in the report that is produced. Also, the seed is included in the SVEM script that you can save to the data table. See the Report Options page.

Warning

A random seed in JMP 14 will not necessarily produce the same analysis as in later versions of JMP.

Proportion of Runs to Exclude

In the Proportion of Runs to Exclude field, enter a value between 0 and 1 representing the proportion of the worst bootstrap runs to exclude from the final model. The default value is 0.5, meaning that half the bootstrap runs will be excluded.

If the Leave K Out setting is turned on, the out-of-bag root average square error or OOB RASE (see Leave K Out) is used to determine the worst bootstrap runs to exclude. That is, the specified proportion of bootstrap runs with the largest OOB RASE values are excluded. If the Leave K Out setting is turned off, validation RASE is used instead.

The SVEM algorithm generates an ensemble or collection of models using fractionally weighted bootstrapping that are combined to form an overall predictive model. Because of the inherent instability of all model fitting algorithms, these models generally vary considerably in terms of OOB RASE or validation RASE. The Proportion of Runs to Exclude strategy provides smaller and smoother prediction error estimates over the factor space.

Leave K Out

The Leave K Out checkbox is checked by default and a value of $K = 2$ is set for the Number of out-of-bag rows, K. You can specify a different positive integer for $K$. You can also uncheck Leave K Out, though this is not recommended. Using Leave K Out yields a more realistic estimate of prediction error because it assesses interpolation between training set observations. It also provides protection against overfitting and reduces the impact of outliers on fitted models.

For each bootstrap run, $K$ random rows (and their corresponding autovalidation rows) are held out of the fitting procedure. These are the out-of-bag (OOB) rows. For $K = 1, 2, \text{or}\:3$, the OOB rows are chosen such that combinations of rows are not repeated until all combinations have been exhausted in previous bootstrap runs. Otherwise, the rows are uniformly sampled without replacement.

If Leave K Out is checked and $K > 0$, then the OOB root average squared error (OOB RASE) for the set of bootstrap runs is reported. This is computed as follows. For each bootstrap run, the averaged squared error is calculated only on the out of bag rows (the OOB error). The OOB RASE shown on the final report is the average of these values across all bootstrap runs. The OOB RASE replaces the validation RASE when calculating standard error.

Note that, for each bootstrap run, the auto-validation set is used as a holdout set to estimate the bootstrap model. The validation RASE measures the model error for this auto-validation set. This error is dependent on the observations and weights used to estimate the model. In contrast, out-of-bag error gives a completely independent estimate of error because it is based on a model derived with the out-of-bag points left out. As such, OOB RASE gives a realistic estimate of prediction error. For this reason, using the Leave K Out option is strongly encouraged.

Number of out-of-bag RASE tours

With the Leave K Out option turned on, the Number of out-of-bag RASE tours option becomes available. The default Number of out-of-bag RASE tours is set to 5.

Response surfaces for physical systems are often complex, reflecting the significant interactivity of these systems. Consequently, finding the best-fitting model can be challenging. Additionally, objective functions in machine learning may not be smooth, further complicating model selection.

Neural networks are fit based on a random selection of a starting point. With complex response surfaces, a random starting point might be located in a suboptimal part of the space, leading to a poor model. A common strategy to enhance the likelihood of selecting better-fitting models is to fit multiple models based on various starting points. Each such model fit is known as a tour.

For each bootstrap iteration, the specified number of tours (or random starts) is utilized to search for the model with the smallest out-of-bag prediction error (OOB RASE). Both simulation studies and evaluations on case studies have demonstrated that the OOB Tours strategy enhances prediction accuracy and reduces prediction error (RASE) in the final ensemble model.

Warning

Based on simulation studies and computation time, the default value is set at 5; however, it should be noted that the tours strategy increases running time so users should be careful in selecting a large value for the number of tours.

Note

It is important to note that the Tours strategy in JMP Pro is based on validation error rather than OOB error. Our findings indicate that the OOB approach offers superior prediction performance. The OOB Tours option is compatible with both JMP standard and JMP Pro.

Set the Number of Hidden Nodes

In the Number of Hidden Nodes panel box, enter an integer value representing the number of TanH nodes to use in the underlying neural network.

Optionally, you can enter a comma-delimited list of integer values and/or ranges in the input field. For each number in the list, SVEM will be fit with the specified number(s) of hidden nodes. For example, when you define 3,5-7,9 as the input integer values, SVEM will be fit with 3, 5, 6, 7, and 9 nodes. All the fit results will be displayed on the same report.

Running SVEM

To run SVEM and generate the results in a report, click Go.

While SVEM is conducting the bootstrap runs, a progress dialog is displayed.

Notes

To stop the run before the specified number of bootstrapped models has been fit for each response, click the Stop Early button. The resulting report will display only the responses that have completed or partially completed their bootstrap runs at the stop time. If you close the progress dialog manually, SVEM continues running until the specified number of bootstrap runs has been completed.

Also, for some larger models, the Stop Early button will not be accessible. In this case, the only way to stop SVEM from running is to press the Esc key repeatedly. The results from the run will not be reported.

After SVEM has completed the run, the SVEM report window is displayed. The SVEM report window is described in more detail in the following section.

SVEM Report Window

The SVEM report window displays the following reports.

Report Type	Example Screenshot
The Actual by Predicted plot shows observed values on the vertical axis and predicted values on the horizontal axis. If the model provides a good fit, the points are positioned close to the diagonal line y = x. Note: Not shown for categorical responses.
The Model Summary report provides details about the model fitting process, including the SVEM settings, and metrics for the final ensemble model. The random seed is provided in case you want to reproduce your analysis. The final model root average squared error (RASE) is computed from the training data, but because of the inherent self-validation capability of SVEM, it is also considered a validation metric.
The Residual by Predicted plot shows the actual values minus their predicted values. Note: Not shown for categorical responses.
For categorical responses, a Receiver-Operating Characteristic (ROC) plot is shown. If the response is Nominal, as in the top plot on the right, the ROC plot shows the sensitivity and (1 - specificity) trade-off for each level of the response. Area Under the Curve (AUC) values for each level of the response are shown parenthetically in the legend of the plot. If the response is Ordinal with J levels, the ROC plot shows the sensitivity and (1 - specificity) trade-off for J-1 levels of the response. For each level shown, the ROC curve is constructed using response values greater than or equal to that level as "positives." There is no plot for the lowest level in the value ordering because all responses are greater than or equal to that level. Area Under the Curve (AUC) values for each level of the response are shown parenthetically in the legend of the plot. The bottom plot on the right is for an ordinal response with levels Low < Medium < High; in this example, J=3, and the two levels shown are Medium and High. The plot labeled Medium includes responses of both Medium and High (i.e, responses that are at least Medium) as positives.

Report Options

Report options are available in the SVEM report window. For details, see the Report Options page.