Description of Experiments
Three different approaches were tested to assess and evaluate better the Sen4CAP classification:
Experiment 1 – New Categories:
In the context of this experiment, we decided to merge all the available crop categories into single classes according to the physiology, genus and characteristics of crops. This can help the classification model as finding a more generic pattern for each new crop category is easier than looking at the characteristic for more than 100 crop categories. For instance, plots of traditional vineyards, table vineyards and wine vineyards are merged into a new category named Vines.
Experiment 2 – No Fallow Land:
Fallow land sometimes in crop classification/mapping tasks can be tricky and can change the results and, as an outcome, lower Overall Accuracy and Kappa coefficient. The problem with plots registered as “Fallow Land” is that farmers sometimes tend to go against the regulations, and despite registering their plots as Fallow Land, they continue with cultivation practices, which is confusing for the model. More details about the Fallow Land results can be found in section Fallow Land detection.
Experiment 3 – Merged Categories:
After executing the first two experiments, we noticed that some categories had very low accuracy (green manure, leguminous, other crops and flowers) or could be merged into a new category, so we created the new category as they have similar characteristics to examine if the model would be more accurate in the classification. This was performed having in mind the that the Basic Payment Scheme treats these crop types equally
Experiment results
The following table present the results of the three experiments performed using Sen4CAP. Two primary metrics were used to evaluate the experiment results, the overall accuracy, and the Kappa coefficient. Overall accuracy is defined as the fraction of the total corrected classified parcels on all categories divided by the total number of parcels. The Kappa coefficient indicates how closely the parcels classified by the used classifier matched the data labelled as ground truth. The table below, presents the overall accuracy and the Kappa coefficient for each one of the experiments.
Experiments
1. 67.96% (Overall accuracy) , 50.51% (Kappa coefficient)
2. 79.37% (Overall accuracy) , 60,35% (Kappa coefficient)
3. 70.43% (Overall accuracy) , 50.97% (Kappa coefficient)
Experiment 1 – Accuracy Metrics
Experiment 2 – Accuracy Metrics
Experiment 3 – Accuracy Metrics
[1] Tree Crops are changing crop code from 12 to 11 for Experiment 3 due to merging different categories.
Green = Higher accuracy
Yellow = Medium accuracy
Orange = Lower accuracy
Gray = Equal/ Almost Equal accuracy
Red = Not included category in the experiment
To better understand the results, two sites were chosen to visualise the differences between the three experiments. As indicated in figure below, example area 1 is located near Paphos Airport, and example area 2 is found in the Akrotiri area. After excluding the Fallow land (Experiment 2), it is evident that the parcels that are classified correctly are more compared to those in Experiments 1 and 3, with Experiment 3 performing slightly better than Experiment 1.
