3. Run a evaluation function with the selected tools and test data sets
The prcbench package currently provides two different
types of performance evaluation.
Accuracy evaluation of precision-recall curves
Running-time evaluation of precision-recall curves
3a. Accuracy evaluation of precision-recall curves
The run_evalcurve function evaluates precision-recall
curves with the following five test cases. The basic idea is that the
function returns the full score as long as the points generated by a
library matches with the manually calculated recall and precision
values.
| Test case | Description |
|---|---|
| fpoint | Check the first point |
| int_pts | Check the intermediate points |
| epoint | Check the end point |
| x_range | Evaluate a range between two recall values |
| y_range | Evaluate a range between two precision values |
Evaluation scores
The run_evalcurve function calculates the scores of the
test cases and summarizes them to a data frame.
## Evaluate precision-recall curves for ROCR and precrec with c1 test set
testset <- create_testset("curve", "c1")
toolset <- create_toolset(c("ROCR", "precrec"))
scores <- run_evalcurve(testset, toolset)
scores## testset toolset toolname score
## 1 c1 ROCR ROCR 5/8
## 2 c1 precrec precrec 8/8The result of each test case can be displayed by specifying
data_type = all of the print
function.
## testset toolset toolname testitem testcat success total
## 1 c1 ROCR ROCR x_range Rg 1 1
## 2 c1 ROCR ROCR y_range Rg 1 1
## 3 c1 ROCR ROCR fpoint SE 0 1
## 4 c1 ROCR ROCR intpts Ip 2 4
## 5 c1 ROCR ROCR epoint SE 1 1
## 6 c1 precrec precrec x_range Rg 1 1
## 7 c1 precrec precrec y_range Rg 1 1
## 8 c1 precrec precrec fpoint SE 1 1
## 9 c1 precrec precrec intpts Ip 4 4
## 10 c1 precrec precrec epoint SE 1 1Visualization of the result
The autoplot shows a plot with the result of the
run_evalcurve function.
## ggplot2 is necessary to use autoplot
library(ggplot2)
## Plot base points and the result of precrec on c1, c2, and c3 test sets
testset <- create_testset("curve", c("c1", "c2", "c3"))
toolset <- create_toolset("precrec")
scores1 <- run_evalcurve(testset, toolset)
autoplot(scores1)
## Plot the results of PerfMeas and PRROC on c1, c2, and c3 test sets
toolset <- create_toolset(c("PerfMeas", "PRROC"))
scores2 <- run_evalcurve(testset, toolset)
autoplot(scores2, base_plot = FALSE)
3b. Running-time evaluation of precision-recall curves
The run_benchmark function internally calls the
microbenchmark function provided by the microbenchmark
package. It takes a test set and a tool set and returns the result of
microbenchmark.
## Run microbenchmark for aut5 on b10
testset <- create_testset("bench", "b10")
toolset <- create_toolset(set_names = "auc5")
res <- run_benchmark(testset, toolset)
res## testset toolset toolname min lq mean median uq max neval
## 1 b10 auc5 AUCCalculator 2.235 2.778 5.372 3.053 3.67 15.12 5
## 2 b10 auc5 PRROC 0.166 0.170 0.200 0.176 0.19 0.30 5
## 3 b10 auc5 PerfMeas 0.067 0.069 0.095 0.071 0.08 0.19 5
## 4 b10 auc5 ROCR 1.792 1.821 1.939 1.845 2.01 2.23 5
## 5 b10 auc5 precrec 4.423 4.476 4.590 4.513 4.55 4.99 5
