eval_scripts package
Submodules
eval_scripts.complexity module
- eval_scripts.complexity.all_in_one_test(testprocess, testmodules)
- eval_scripts.complexity.all_in_one_train(trainprocess, trainmodules)
- eval_scripts.complexity.getallparams(li)
eval_scripts.performance module
- eval_scripts.performance.AUPRC(pts)
- eval_scripts.performance.accuracy(truth, pred)
- eval_scripts.performance.eval_affect(truths, results, exclude_zero=True)
- eval_scripts.performance.f1_score(truth, pred, average)
- eval_scripts.performance.ptsort(tu)
eval_scripts.robustness module
- eval_scripts.robustness.effective_robustness(robustness_result, task)
Compute the effective robustenss metric given the performance of the method on the task.
- eval_scripts.robustness.effective_robustness_helper(robustness_result, task)
Helper function that computes the effective robustness metric as the performance difference compared to late fusion method.
- Parameters:
robustness_result – Performance of the method on datasets applied with different level of noises.
task – Name of the task on which the method is evaluated.
- eval_scripts.robustness.get_robustness_metric(robustness_result, task, metric)
Compute robustness metric given specific method performance and the task.
- Parameters:
robustness_result – Performance of the method on datasets applied with different level of noises.
task – Name of the task on which the method is evaluated.
metric – Type of robustness metric to be computed. ( “effective” / “relative” )
- eval_scripts.robustness.maxmin_normalize(result, task)
Normalize the metric for robustness comparison across all methods.
- Parameters:
result – Un-normalized robustness metrics of all methods on the given task.
task – Name of the task.
- eval_scripts.robustness.relative_robustness(robustness_result, task)
Compute the relative robustenss metric given the performance of the method on the task.
- eval_scripts.robustness.relative_robustness_helper(robustness_result, task)
Helper function that computes the relative robustness metric as the area under the performance curve.
- Parameters:
robustness_result – Performance of the method on datasets applied with different level of noises.
- eval_scripts.robustness.single_plot(robustness_result, task, xlabel, ylabel, fig_name, method)
Produce performance vs. robustness plot of a single method.
- Parameters:
robustness_result – Performance of the method on dataset applied with different level of noises.
task – Name of the task on which the method is evaluated.
xlabel – Label of x-axis to be appeared in the plot.
ylabel – Label of y-axis to be appeared in the plot.
fig_name – Name of plot to be saved.
method – Name of the method.