Once you have created a model you are satisfied with, you should test it on data other than the data collection used to create it. This will ensure your model works on other scans as well, and will give you more confidence regarding its robustness.
To test your model’s performance, simply click test model and select one of the other collections in your account and click test. The model will analyze each scan in the test collection, and the results will be downloaded as a text file to your computer so you can do whatever research you want on it (for example, finding the correlation between high errors and one of the attributes, estimating the performance of the outlier detection algorithm, etc.)

screenshot-1

 

screenshot-2

 

screenshot-3

When you create a classification model, scan responses are related to one of the classes in the data collection. For example, if you scan only hand and desk when creating a model, when testing the model you will always get one of these classes, even if you scan something completely different (for example, orange).

The Outlier Detection feature will produce an additional scan response of ‘null’ for unrecognized materials.

Outlier Detection Outlier Detection

If you activate the Outlier Detection mode before you create a model, when testing your model on unrecognized material the response will be “null”.

To activate Outlier Detection mode:

  • Set the expert mode to On (expert mode is set as off by default)
  • Click the cog wheel
  • In Settings, check the Outlier Detection checkbox
Each time you build a successful model, SCiO Lab provides performance figures on the model.
reading models performance
In this case, R2 is the correlation between the estimation and the true values. For more information about R2, click here.
RMSE is the root mean square error (in a sense, the typical error).
A perfect model will have R2=1 and RMSE = 0.

When you assess the performance of an estimation model you’ll want to use the Known vs. Estimated scatter plot.

In an ideal situation your model estimated value is the same as the known value. But the reality is that noise and other effects can influence your results and this does not occur. To analyze the success or failure of your potential model, SCiO Lab enables you to view the results as a scatter plot.

To create a scatter plot:

    1. Navigate to the Spectrum tab of a Data Collection
    2. Select an attribute to filter the model building from using the Analyze by filter on the left side of the screen.

      1. Apply one of the pre-designed preprocessing methods or create your own using the expert mode.
      2. Click Create Model.
      3. Your model’s performance is displayed as a Known vs. Estimated Scatter Plot, and provides performance metrics and if needed, suggestions for improvement. If you model does not contain enough samples, a warning will appear above your Scatter Plot, alerting you to scan more before trying to build.

 

Good Model_known vs estimated

Good Scatter Plot

Bad Model_known vs estimated

Bad scatter plot

On each scatter plot, the horizontal axis shows the known values and the vertical axis displays the estimated values. Each point on the scatter plot represents a data point (one scan). In the scatter plot, the central line represents the optimal model. The upper and lower limit lines represent the 20% error margin.

Once you have collected enough samples to consider your Data Collection complete, now it is time to test your model. Your Models will be used as the basis of your future SCiO Applications. If you are a SCiO Developer that only wants to build mobile applications, but you don’t want to create you own models, don’t worry! You’ll be able to build your apps on top of the SCiO Lab Models we release and share.

For more information, see Observing Data and Data Scrubbing.