mistk.evaluation.abstract_evaluation_plugin module

class mistk.evaluation.abstract_evaluation_plugin.AbstractEvaluationPlugin

Bases: object

classdocs

assessment_types()

Assessment types that are supported by the evaluate method

abstract do_evaluate(assessment_type, metrics, input_data_format, evaluation_input_format, ground_truth_path, evaluation_path, properties)

Performs metrics’ evaluation using the predictions and ground truth files provided. Stored the assessment results as a JSON file in the evaluation_path

Parameters

  • assessment_type – The evaluation type. One of {‘BinaryClassification’, ‘MultilabelClassification’, ‘MulticlassClassification’, ‘Regression’}

  • metrics – Specific metrics to evaluate against instead of all metrics defined by assessment_type

  • input_data_path – Path to input data for the evaluation

  • evaluation_input_format – The format of the input data

  • ground_truth_path – The directory path where the ground_truth.csv file is located

  • evaluation_path – A directory path to where the evaluation.json output file will be stored

  • properties – A dictionary of key value pairs for evaluation plugin arguments.

abstract do_terminate()

Stops all processing and releases any resources that are in use in preparation for being shut down.

property endpoint_service

Returns the endpoint service for this Evaluation

evaluate(assessment_type, metrics, input_data_path, evaluation_input_format, ground_truth_path, evaluation_path, properties)

Triggers the model to enter the evaluation state. A subsequent call to do_evaluate with the given parameters will be made as a result.

This method should not be implemented or overwritten by subclasses. It will be created by the state machine.

fail(reason)

Triggers the model to enter the failed state. This method should not be implemented or overwritten by subclasses. It will be created by the state machine.

Parameters

reason – The reason why the state machine is transitioned into the failure state

metrics()

Metrics that can be performed by the evaluate method

new_state_entered(*args, **kwargs)

Notifies the endpoint service that the current state of the state machine has been update

Parameters

  • args – Optional non-keyworded variable length arguments to pass in

  • kwargs – Optional keyworded variable length arguments to pass in

ready()

Triggers the model to enter the ready state. It is expected that this method will only be called by the implementing model subclass.

This method should not be implemented or overwritten by subclasses. It will be created by the state machine.

report_failure(reason)

Reports the failure of a model during its operations

Parameters

reason – The error message explaining why the model failed.

terminated()

Shutdowns the evaluation plugin

update_status(payload)

Provides additional details to the current state of the evaluation.

Parameters

payload – Extra information regarding the status update