CrossValidator (Spark 2.3.4 JavaDoc)

Object
- org.apache.spark.ml.PipelineStage
- - org.apache.spark.ml.Estimator<CrossValidatorModel>
  - - org.apache.spark.ml.tuning.CrossValidator

All Implemented Interfaces:

java.io.Serializable, Logging, Params, HasCollectSubModels, Identifiable, MLWritable
```
public class CrossValidator
extends Estimator<CrossValidatorModel>
implements HasCollectSubModels, MLWritable, Logging
```
K-fold cross validation performs model selection by splitting the dataset into a set of non-overlapping randomly partitioned folds which are used as separate training and test datasets e.g., with k=3 folds, K-fold cross validation will generate 3 (training, test) dataset pairs, each of which uses 2/3 of the data for training and 1/3 for testing. Each fold is used as the test set exactly once.

See Also:

Serialized Form

Constructor Summary

Constructors
Constructor and Description

CrossValidator()

CrossValidator(String uid)

Constructors
Constructor and Description
`CrossValidator()`
`CrossValidator(String uid)`

Method Summary

All Methods Static Methods Instance Methods Abstract Methods Concrete Methods
Modifier and Type	Method and Description
`static Params`	`clear(Param<?> param)`
`static BooleanParam`	`collectSubModels()`
`CrossValidator`	`copy(ParamMap extra)` Creates a copy of this instance with the same UID and some extra params.
`static Param<Estimator<?>>`	`estimator()`
`Param<Estimator<?>>`	`estimator()` param for the estimator to be validated
`static Param<ParamMap[]>`	`estimatorParamMaps()`
`Param<ParamMap[]>`	`estimatorParamMaps()` param for estimator param maps
`static Param<Evaluator>`	`evaluator()`
`Param<Evaluator>`	`evaluator()` param for the evaluator used to select hyper-parameters that maximize the validated metric
`static String`	`explainParam(Param<?> param)`
`static String`	`explainParams()`
`static ParamMap`	`extractParamMap()`
`static ParamMap`	`extractParamMap(ParamMap extra)`
`CrossValidatorModel`	`fit(Dataset<?> dataset)` Fits a model to the input data.
`static <T> scala.Option<T>`	`get(Param<T> param)`
`static boolean`	`getCollectSubModels()`
`static <T> scala.Option<T>`	`getDefault(Param<T> param)`
`static Estimator<?>`	`getEstimator()`
`Estimator<?>`	`getEstimator()`
`static ParamMap[]`	`getEstimatorParamMaps()`
`ParamMap[]`	`getEstimatorParamMaps()`
`static Evaluator`	`getEvaluator()`
`Evaluator`	`getEvaluator()`
`static int`	`getNumFolds()`
`int`	`getNumFolds()`
`static <T> T`	`getOrDefault(Param<T> param)`
`static int`	`getParallelism()`
`static Param<Object>`	`getParam(String paramName)`
`static long`	`getSeed()`
`static <T> boolean`	`hasDefault(Param<T> param)`
`static boolean`	`hasParam(String paramName)`
`static boolean`	`isDefined(Param<?> param)`
`static boolean`	`isSet(Param<?> param)`
`static CrossValidator`	`load(String path)`
`void`	`logTuningParams(<any> instrumentation)` Instrumentation logging for tuning params including the inner estimator and evaluator info.
`static IntParam`	`numFolds()`
`IntParam`	`numFolds()` Param for number of folds for cross validation.
`static IntParam`	`parallelism()`
`static Param<?>[]`	`params()`
`static MLReader<CrossValidator>`	`read()`
`static void`	`save(String path)`
`static LongParam`	`seed()`
`static <T> Params`	`set(Param<T> param, T value)`
`CrossValidator`	`setCollectSubModels(boolean value)` Whether to collect submodels when fitting.
`CrossValidator`	`setEstimator(Estimator<?> value)`
`CrossValidator`	`setEstimatorParamMaps(ParamMap[] value)`
`CrossValidator`	`setEvaluator(Evaluator value)`
`CrossValidator`	`setNumFolds(int value)`
`CrossValidator`	`setParallelism(int value)` Set the maximum level of parallelism to evaluate models in parallel.
`CrossValidator`	`setSeed(long value)`
`static String`	`toString()`
`StructType`	`transformSchema(StructType schema)` :: DeveloperApi ::
`StructType`	`transformSchemaImpl(StructType schema)`
`String`	`uid()` An immutable unique ID for the object and its derivatives.
`MLWriter`	`write()` Returns an `MLWriter` instance for this ML instance.

Methods inherited from class org.apache.spark.ml.Estimator
fit, fit, fit, fit

Methods inherited from class Object
equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.spark.ml.param.shared.HasSeed
getSeed, seed

Methods inherited from interface org.apache.spark.ml.param.Params
clear, copyValues, defaultCopy, defaultParamMap, explainParam, explainParams, extractParamMap, extractParamMap, get, getDefault, getOrDefault, getParam, hasDefault, hasParam, isDefined, isSet, paramMap, params, set, set, set, setDefault, setDefault, shouldOwn

Methods inherited from interface org.apache.spark.ml.util.Identifiable
toString

Methods inherited from interface org.apache.spark.ml.param.shared.HasCollectSubModels
collectSubModels, getCollectSubModels

Methods inherited from interface org.apache.spark.ml.util.MLWritable
save

Methods inherited from interface org.apache.spark.internal.Logging
initializeLogging, initializeLogIfNecessary, initializeLogIfNecessary, isTraceEnabled, log_, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning

- Constructor Detail
  - CrossValidator
```
public CrossValidator(String uid)
```
  - CrossValidator
```
public CrossValidator()
```
- Method Detail
  - read
```
public static MLReader<CrossValidator> read()
```
  - load
```
public static CrossValidator load(String path)
```
  - toString
```
public static String toString()
```
  - params
```
public static Param<?>[] params()
```
  - explainParam
```
public static String explainParam(Param<?> param)
```
  - explainParams
```
public static String explainParams()
```
  - isSet
```
public static final boolean isSet(Param<?> param)
```
  - isDefined
```
public static final boolean isDefined(Param<?> param)
```
  - hasParam
```
public static boolean hasParam(String paramName)
```
  - getParam
```
public static Param<Object> getParam(String paramName)
```
  - set
```
public static final <T> Params set(Param<T> param,
                                   T value)
```
  - get
```
public static final <T> scala.Option<T> get(Param<T> param)
```
  - clear
```
public static final Params clear(Param<?> param)
```
  - getOrDefault
```
public static final <T> T getOrDefault(Param<T> param)
```
  - getDefault
```
public static final <T> scala.Option<T> getDefault(Param<T> param)
```
  - hasDefault
```
public static final <T> boolean hasDefault(Param<T> param)
```
  - extractParamMap
```
public static final ParamMap extractParamMap(ParamMap extra)
```
  - extractParamMap
```
public static final ParamMap extractParamMap()
```
  - seed
```
public static final LongParam seed()
```
  - getSeed
```
public static final long getSeed()
```
  - estimator
```
public static Param<Estimator<?>> estimator()
```
  - getEstimator
```
public static Estimator<?> getEstimator()
```
  - estimatorParamMaps
```
public static Param<ParamMap[]> estimatorParamMaps()
```
  - getEstimatorParamMaps
```
public static ParamMap[] getEstimatorParamMaps()
```
  - evaluator
```
public static Param<Evaluator> evaluator()
```
  - getEvaluator
```
public static Evaluator getEvaluator()
```
  - numFolds
```
public static IntParam numFolds()
```
  - getNumFolds
```
public static int getNumFolds()
```
  - parallelism
```
public static IntParam parallelism()
```
  - getParallelism
```
public static int getParallelism()
```
  - collectSubModels
```
public static final BooleanParam collectSubModels()
```
  - getCollectSubModels
```
public static final boolean getCollectSubModels()
```
  - save
```
public static void save(String path)
                 throws java.io.IOException
```
    Throws:
    
    java.io.IOException
  - uid
```
public String uid()
```
    Description copied from interface: Identifiable
    
    An immutable unique ID for the object and its derivatives.
    
    Specified by:
    
    uid in interface Identifiable
    
    Returns:
    
    (undocumented)
  - setEstimator
```
public CrossValidator setEstimator(Estimator<?> value)
```
  - setEstimatorParamMaps
```
public CrossValidator setEstimatorParamMaps(ParamMap[] value)
```
  - setEvaluator
```
public CrossValidator setEvaluator(Evaluator value)
```
  - setNumFolds
```
public CrossValidator setNumFolds(int value)
```
  - setSeed
```
public CrossValidator setSeed(long value)
```
  - setParallelism
```
public CrossValidator setParallelism(int value)
```
    Set the maximum level of parallelism to evaluate models in parallel. Default is 1 for serial evaluation
    
    Parameters:
    
    value - (undocumented)
    
    Returns:
    
    (undocumented)
  - setCollectSubModels
```
public CrossValidator setCollectSubModels(boolean value)
```
    Whether to collect submodels when fitting. If set, we can get submodels from the returned model.
    Note: If set this param, when you save the returned model, you can set an option "persistSubModels" to be "true" before saving, in order to save these submodels. You can check documents of CrossValidatorModel.CrossValidatorModelWriter for more information.
    
    Parameters:
    
    value - (undocumented)
    
    Returns:
    
    (undocumented)
  - fit
```
public CrossValidatorModel fit(Dataset<?> dataset)
```
    Description copied from class: Estimator
    
    Fits a model to the input data.
    
    Specified by:
    
    fit in class Estimator<CrossValidatorModel>
    
    Parameters:
    
    dataset - (undocumented)
    
    Returns:
    
    (undocumented)
  - transformSchema
```
public StructType transformSchema(StructType schema)
```
    Description copied from class: PipelineStage
    
    :: DeveloperApi ::
    Check transform validity and derive the output schema from the input schema.
    We check validity for interactions between parameters during transformSchema and raise an exception if any parameter value is invalid. Parameter value checks which do not depend on other parameters are handled by Param.validate().
    Typical implementation should first conduct verification on schema change and parameter validity, including complex parameter interaction checks.
    
    Specified by:
    
    transformSchema in class PipelineStage
    
    Parameters:
    
    schema - (undocumented)
    
    Returns:
    
    (undocumented)
  - copy
```
public CrossValidator copy(ParamMap extra)
```
    Description copied from interface: Params
    
    Creates a copy of this instance with the same UID and some extra params. Subclasses should implement this method and set the return type properly. See defaultCopy().
    
    Specified by:
    
    copy in interface Params
    
    Specified by:
    
    copy in class Estimator<CrossValidatorModel>
    
    Parameters:
    
    extra - (undocumented)
    
    Returns:
    
    (undocumented)
  - write
```
public MLWriter write()
```
    Description copied from interface: MLWritable
    
    Returns an MLWriter instance for this ML instance.
    
    Specified by:
    
    write in interface MLWritable
    
    Returns:
    
    (undocumented)
  - numFolds
```
public IntParam numFolds()
```
    Param for number of folds for cross validation. Must be >= 2. Default: 3
    
    Returns:
    
    (undocumented)
  - getNumFolds
```
public int getNumFolds()
```
  - estimator
```
public Param<Estimator<?>> estimator()
```
    param for the estimator to be validated
    
    Returns:
    
    (undocumented)
  - getEstimator
```
public Estimator<?> getEstimator()
```
  - estimatorParamMaps
```
public Param<ParamMap[]> estimatorParamMaps()
```
    param for estimator param maps
    
    Returns:
    
    (undocumented)
  - getEstimatorParamMaps
```
public ParamMap[] getEstimatorParamMaps()
```
  - evaluator
```
public Param<Evaluator> evaluator()
```
    param for the evaluator used to select hyper-parameters that maximize the validated metric
    
    Returns:
    
    (undocumented)
  - getEvaluator
```
public Evaluator getEvaluator()
```
  - transformSchemaImpl
```
public StructType transformSchemaImpl(StructType schema)
```
  - logTuningParams
```
public void logTuningParams(<any> instrumentation)
```
    Instrumentation logging for tuning params including the inner estimator and evaluator info.
    
    Parameters:
    
    instrumentation - (undocumented)

Class CrossValidator

Constructor Summary

Method Summary

Methods inherited from class org.apache.spark.ml.Estimator

Methods inherited from class Object

Methods inherited from interface org.apache.spark.ml.param.shared.HasSeed

Methods inherited from interface org.apache.spark.ml.param.Params

Methods inherited from interface org.apache.spark.ml.util.Identifiable

Methods inherited from interface org.apache.spark.ml.param.shared.HasCollectSubModels

Methods inherited from interface org.apache.spark.ml.util.MLWritable

Methods inherited from interface org.apache.spark.internal.Logging

Constructor Detail

CrossValidator

CrossValidator

Method Detail

read

load

toString

params

explainParam

explainParams

isSet

isDefined

hasParam

getParam

set

get

clear

getOrDefault

getDefault

hasDefault

extractParamMap

extractParamMap

seed

getSeed

estimator

getEstimator

estimatorParamMaps

getEstimatorParamMaps

evaluator

getEvaluator

numFolds

getNumFolds

parallelism

getParallelism

collectSubModels

getCollectSubModels

save

uid

setEstimator

setEstimatorParamMaps

setEvaluator

setNumFolds

setSeed

setParallelism

setCollectSubModels

fit

transformSchema

copy

write

numFolds

getNumFolds

estimator

getEstimator

estimatorParamMaps

getEstimatorParamMaps

evaluator

getEvaluator

transformSchemaImpl

logTuningParams