Development of a ﬂexible Computer Vision System for marbling 1 classiﬁcation

10 Traditional marbling meat evaluation is a tedious, repetitive, costly and time-consuming task 11 performed by panellists. Alternatively, we have Computer Vision Systems (CVS) to mitigate 12 these problems. However, most of CVS are restricted to speciﬁc environments, conﬁgurations 13 or muscle types, and marbling scores are settled for a particular marbling meat standard. 14 In this context, we developed a CVS for meat marbling grading, which is ﬂexible to diﬀerent 15 muscle colour contrasts and grading standards. Essentially, the proposed method segments 16 an image pre-processed by illumination normalisation and contrast enhancement, analyses 17 visible intramuscular fat pixels and attributes a score based on a desired meat standard 18 deﬁned in the learning step. Learning approach is an instance-based system making use 19 of k -Nearest Neighbours algorithm ( k -NN) to attribute a score from segmentation results. 20 The algorithm classiﬁes the new samples based on scores assigned by panellists. We in-21 vestigated the optimal number of samples for modelling, focusing on the smallest number 22 leading to acceptable accuracy, and considering two diﬀerent animal species: bovine and 23


Introduction
Traditional evaluation process of meat quality is tedious, laborious, highly repetitive, costly, time-consuming and requires trained specialists (Sun, 2011(Sun, , 2012;;Qiao et al., 2007;Chen and Qin, 2008;Jackman et al., 2009;Liu et al., 2012;Huang et al., 2013).Several studies have highlighted marbling as an important meat quality parameter; however, the traditional evaluation approaches can be influenced by the subjective visual and sensory criteria adopted by the involved specialists (Xiong et al., 2014).
Marbling consists in visible portions of intramuscular fat and it influences other meat attributes such as tenderness, flavor and texture.Furthermore, marbling level influences consumers choice, since a high marbling degree indicates a superior meat quality (Faucitano et al., 2005;Killinger et al., 2004).
In general, specialists determine marbling scores based on a visual assessment supported by standard meat images.Meat standards are labelled according to numerical scales related to the visible amount of intramuscular fat.Several standards have been defined for marbling classification according to country, meat type and animal species, such as the Japanese standard, the Australian standard, the Canadian standard and the USDA standard.Therefore, a generalised approach capable of handling various types of meat and different standard scales could constitute a valuable help to facilitate meat marbling assessment (Cheng et al., 2015).Liu et al. (2012) described several research works focused on objective marbling assessment in specific species, mainly dealing with colour contrast differences.Several Computer Vision Systems (CVS) were proposed, that are however designed for specific marbling standards and animal species.This implies that various parameters and thresholds need to be tuned depending on the specific problem at hand, i.e., based on the considered marbling standard and on the particular animal species that is evaluated.
CVS has been widely used in food industry for food quality evaluation and control (Jackman et al., 2012).Marbling assessment can be performed by a CVS by means of a digital camera, which is inexpensive and widely available.However, a general approach that can lead to good results independently of muscle colour, contrast, standard or species is still a challenge.This challenge can be tackled using a CVS approach combined with machine learning algorithms.In (Qiao et al., 2007;Jackman et al., 2009;Huang et al., 2013), CVS has been used for marbling assessment, leading to satisfactory results.However, these works report expensive solutions based on quite controlled environments, costly equipments and parametrised algorithms for image processing.
In Jackman et al. (2009) a marbling segmentation algorithm has been proposed.Actually, in this paper the Authors did not calculate a marbling score after the segmentation phase.However, they suggested the use of artificial intelligence based processes that could learn from the panellists assessments, in order to gain more advanced levels of adaptability.Furthermore, in this paper as well as in other research works dealing with similar issues (Chen and Qin, 2008;Peña et al., 2013), the image acquisition step requires to consider a controlled environment, often using specific camera models and configurations (exposure compensation, aperture, lens and ISO).These issues cause difficulties in CVS reproduction and industrial application.Some CVS need to deal with nonlinearities between the image features and the marbling score of interest, making use of sophisticated modelling techniques from artificial intelligence.Furthermore, specific parametrisation and thresholds could lead to scarcely reproducible solutions.Thus, in order to implement a robust CVS able to cope with more complex scenarios, it is recommended to apply machine learning algorithms.Machine Learning (ML) is an effective tool for exploratory data analysis and is widely employed for various applications, including Computer Vision.(Ropodi et al., 2016).
The application of ML algorithms for food evaluation has been widely investigated (Du and Sun, 2006;Balasubramanian et al., 2009;Chen et al., 2010;Valous et al., 2010;Wang et al., 2012;Liu et al., 2013;Papadopoulou et al., 2013;Prevolnik et al., 2014;Muñoz et al., 2015), demonstrating that ML can be applied to uncover non-trivial relationships by automatically learning from a set of training data, thus producing knowledge which in turn can be used to interpret new data.
The choice of the most proper machine learning algorithm is related to its properties and to the set of assumptions used by the learner to estimate the output for those examples that have not been considered in the training phase (which is known as induc-tive bias); these aspects are mainly related to data representation and local-versus-global learning (García et al., 2008).In particular, in the present work we aimed at considering the smallest possible number of instances enabling to predict classes (marbling score values) with acceptable accuracy.For this reason, k -Nearest Neighbour (k -NN) classifier was considered, since it is a simple supervised learning scheme which classifies unknown instances by finding the closest previously observed instances (Brighton and Mellish, 2002).Learners which apply this classification method are named Instance-Based Learners.
O 'Farrell et al. (2005) compared k -NN usage to ANN (Artificial Neural Networks), more precisely to a MLP (Multi-layer Perceptron), in order to verify whether a simple classification technique like k -NN could fit for quality control in food industry.The Authors, citing also several research works on food matrices, concluded that k -NN may be entirely satisfactory and is computationally very simple.In Barbon et al. (2016), the performance of k -NN to predict pork storage time was compared to seven other algorithms (Random Forest, MLP, Support Vector Machine, J48 and Naïve Bayes, and two different Fuzzy methods), leading to the second best accuracy values.
In this context, this paper contributes to the current research in the field by presenting a method to perform marbling grading based on image analysis, designed in a way to be able to handle different muscles of various animal species, and to be adaptable to diverse marbling standards.In particular, our CVS is based on dynamic thresholding, illumination normalisation, adaptive contrast enhancement and instance-based decision for marbling grading.The performance of the proposed method was evaluated considering meat samples from two different animal species (beef and pork), each one with its own marbling standard.

Materials and Methods
The overall proposed method is exhibited in Figure 1, which shows the main stages numbered as 1, 2 and 3. Stage 1 refers to the establishment of desired meat standard and exemplification of each level by tagging some image examples.Details of how we conduct this step and data sets used in experiments are available in Section 2.1.The results of this stage are applied to instance-based modelling and automatic grading of the new samples.
Stage 2 is marbling segmentation kernel, performed by applying a series of image processing steps, which are described in Section 2.2.Finally, Stage 3 (Section 2.3) is focused on the instance-based marbling score by k -NN, regarding advantages of the selected algorithm, how it can be applied and evaluation criteria.

Samples and Panellist Analysis
A requirement of instance-based learning is the availability of labelled instances to perform supervised learning.In other words, some samples must be tagged with appropriate marbling score to build a relation between marbling score and image properties.A panellist performs this task through a conventional approach, which is tedious and time consuming.
For this reason, we proposed the usage of few samples from each grade to reduce the number of samples required for the labelling process.Furthermore, our approach was designed to carry out this stage only once per each standard.In particular, two different muscle foods were considered, each one labelled by the relevant standard scale.Images of pork and beef samples were acquired at 24 hours post mortem and were used to construct the k -NN model and for panellist task.
Image sampling was performed at the Food Analysis Laboratory (LANA), State University of Londrina.The image acquisition setup was placed in an uncontrolled environment, which was illuminated by ambient daylight and cool white fluorescent artificial lighting.
Three hundred thirty-five (335) pork samples and forty-five (45) beef samples were used, both from longissimus thoracis muscle removed between penultimate and last ribs of the left half carcass.Beef samples came from Nelore breed animals, fed on pasture and slaughtered at a federally inspected abattoir.Pork samples came from commercial genetics provided by a local company, and were transported under refrigeration to LANA immediately after slaughtering.
Pork and beef samples images were acquired using a digital single-lens reflex camera, model Nikon SLR D7000 (Nikon Co. Ltd., Japan), equipped with a 16.2 megapixels image sensor and with a high-quality lens, which was optimally engineered to gather more light.
The digital camera was configured with automatic settings.A tripod supported the device at 37cm above samples, which were placed on a blue paper sheet used as image background.
After acquisition and according to Figure 1, pork images were analysed subjectively by experts using traditional marbling methodology based on NPPC photographic standard.
Similarly, all beef images were analysed subjectively by experts following the same methodology used for pork images evaluation, but based on USDA photographic standard.This methodology consists in a subjective analysis based on beef marbling intensity, leading to score values defined according to the following scale: 1 = devoid, 2 = practically devoid, 3 = traces, 4 = slight, 5 = small, 6 = modest, 7 = moderate, 8 = slightly abundant and 9 = moderately abundant (Tan, 2004).
Panellists were trained using digital images, not fresh samples.We consider that assessment based on digital images did not compromise accuracy, since this task was performed as in Tan (2004) and possible distortions or divergences between real and image-based evaluation were avoided by standard based calibration.

Marbling Segmentation
All the image processing steps followed to implement marbling segmentation are shown in Figure 2. The first goal of this marbling segmentation method is background removal, keeping the Region of Interest (ROI) only.To achieve this, red and blue channels (from RGB colour space) of the original image were swapped.According to Jackman et al. (2009), this helps to remove blue backgrounds using image thresholding in Hue channel of HSI (Hue, Saturation and Intensity) colour space.This threshold value was selected using Otsu's method (Otsu, 1979) since it is one of the most accurate and widely used methods for image segmentation (Sahoo et al., 1988).Since this image thresholding step may erroneously lead to the removal of some pixels of the ROI, all the holes in the image were filled using a connectivity approach.
At this point, the obtained image mask is similar to the one reported in Figure 3b, where the blue background region has been removed, but some non-interesting regions are still present.Since the ROIs of our samples were always in the image centre, it was possible to easily remove these non-interesting regions by selecting the central region with a region growing algorithm, leading to an image mask like the one reported in Figure 3c.Once the image ROI was defined, the original image was cropped to fit the ROI.Then, an erosion filter with a disk size equal to 6% of the image dimension was applied to remove the possible presence of fat in the sample border, as it frequently happens both in pork and in beef.
Furthermore, often the imaged samples have dark o light spots, due to sample preparation issues, as it can be seen in Figure 4a.These spots can compromise contrast enhancement methods and also hinder to find a proper threshold value for marbling segmentation.To solve this problem, we applied an illumination normalisation method described in Barbin et al. (2016), which is exemplified in Figure 4.This figure shows that for pork image the illumination normalisation led to a less intense image, while for the beef sample the result-ing image it was possible to observe a intensity enhancement.This aspect can be better appreciated by looking at the intensity (frequency histograms) reported in the top-right of each sub-figure.
Illumination normalisation method starts with a Gaussian blur filtering over a copy of the original image.This action spreads light spots increasing their radius, and creating a gradient of intensity starting from the spots centres.A colour compensation of the blurred image is then performed, so that the spots become darker.The resulting image is then converted to the HSL colour space, and the L (Lightness) channel is selected.In the L image, the intensities of spread light spots are then reversed, so that they can be combined with the original image to attenuate lighter regions.An Overlay blend operation between processed lightness representation and original image is then performed to lead to an illumination normalised image.The Overlay blend is given by equation ( 1): Where E is the resulting image , I is the original image and M is the L channel of the blurred image.As a result, dark regions become darker and light regions become lighter.Based on the processed lightness image, light spots are attenuated, while regions with homogeneous illumination are less changed.Channel Subtraction was then applied to enhance contrast.Using the HSV colour space, the contrast-enhanced image was obtained by subtracting the Saturation (S) channel from the Value (V) channel.The effect of this process can be seen in Figure 5, where figures 5a and 5e are the input images, figures 5b and 5f are the grey-scale input images (for comparison only), and figures 5c and 5g are the contrast enhanced images for pork and meat samples, respectively.The contrast difference between original and enhanced images can be observed also by comparing the frequency histograms at the top-right of figures 5b -5c and of figures 5f -5g for pork and beef, respectively.
By performing illumination normalisation and contrast enhancement steps, the robust-ness of our solution was increased.It made the approach less susceptible to acquisition problems, like colour and light variations or camera settings.
Erosion method was then applied to eliminate fat coverage, by removing the border pixels from the region of interest (Hansard et al. (2014)), as it can be seen in Figures 5d and 5h.At this stage, image is ready for thresholding, which will segment marbling from muscle.
After the thresholding step (max-entropy), small objects (smaller than 0.01% of image's size) were removed to avoid noise, due e.g. to specular reflection, as proposed by Jackman et al. (2009).The effect of thresholding and noise removal on two sample images can be observed in Figure 6 for pork and beef, respectively.
Even though these correction steps during preprocessing may slightly modify the marbling pixels, the machine learning algorithm builds a model able to deal with the modifications caused in the previous stages of our Computer Vision System.
The final result (marbling) can be calculated by the pixel ratio number.For exam-ple, in the case of the pork image reported in Figure 6, this value is calculated as the ratio between the number of pixels of Figure 6c and the number of those of Figure 6a.

Instance-Based Marbling Grading
In this manner, it is possible to quantify for each sample the pixels percentage that represents sample marbling.However, this value is not related to any marbling meat standard model.
As Aggarwal (2014) states, there is a set of algorithms that do not need a complete rebuild in cases instances amount changes.Even when some instances are added to the former dataset, none computational processing would be required.Aggarwal (2014) called them instance-based learners.
Differently from common supervised learning algorithms, instance-based learners do not need a training step to build a model.Instead, all computational effort is focused on classification step.Such characteristic is also a double edge: a) it is possible to change dataset at will, however b) the classification step might be costly (Aggarwal, 2014).
k -NN, an instance-based learner, predicts a sample value by finding its k nearest neighbours.Once k neighbours are found, a mean value is calculated among neighbours and attributed as prediction value to an unknown instance.One advantage of using such algorithm in our solution is that no model is rebuilt as the dataset is updated.In fact, as stated before, no model is returned.
Also, k -NN is very simple and intuitive considering its parameters.Settings of such algorithm include: number of neighbours to be found (k ), metric to be considered to compare neighbours (e.g., Euclidean Distance), and weighted neighbor application 1 2 .
In our approach, Euclidean Distance was used as metric, neighbour weighting was based on 1/distance, and both the number samples, n, and the number of neighbours, k, were optimised in order to find the minimum value leading to acceptable accuracy in classification.In particular, different values of the k parameter of k-NN were tested in the 1 k (n − 1) range, where n is the number of samples considered as a reference for each marbling score value.
k -NN was evaluated by holdout 70/30 stratified with 100 repetitions.Statistical evaluation was performed to evaluate CVS performance and to compare it with human assessment.
1 https://cran.r-project.org/web/packages/FNN/FNN.pdf 2 http://www.mathworks.com/help/stats/classificationk-NN-class.html This evaluation has been performed separately for pork and beef.k -NN from R packages was used in this work, and the results were expressed in terms of Accuracy.

Results and Discussion
Regardless of the analysed species, a panellist took about eleven seconds (11 s) to grade a sample, while CVS can take less than one second (< 1 s) with no breaks.This evaluation corroborate CVS as a solution to tackle a time-consuming task like this one.
The results are presented in the following order: the exploration of the optimal n sample number considered as a reference for each marbling score is reported in subsection 3.1 for pork dataset and in subsection 3.2 for beef dataset.Then, the identification of the best value of the k parameter is discussed in subsection 3.3.Finally, in subsection 3.4 the advantages of the proposed CVS method over other approaches dealing with similar tasks are discussed.

Pork
Results showed that in 100% of images, 335 samples, the maximum absolute difference between CVS score and panellists mean score was lower than one marbling score.
Comparing each marbling score, level one achieved the better accuracy.Figure 7 shows that, using two samples (n = 2) for modelling, marbling score one achieves an average of 90.09% with outliers presence that results in a high standard deviation (0.20).By increasing the n, the average accuracy values of marbling score one were equal to 94.59% (n = 3), 94.32% (n = 4) and 93.57% (n = 5).Using just one sample (n = 1), the average accuracy value of score one resulted equal to only 32.78%.Concerning the accuracy in the estimate of the different marbling score values, in general the best accuracy was obtained for score one, followed by two and three.The score four presented the largest boxplots, across the whole range of samples (n).This occurs due to the fact that the number of available samples with score equal to 4 was lower than the number of samples with the other score values.8a shows a sample ROI which was graded with score 5 by panellist 1 (P1), score 3 by panellist 2 (P2) and score 4 by panellist 3 (P3).After marbling segmentation (Figure 8b), CVS found 2.99% of visible image marbling fat, which corresponds to score 3 according to our K -NN model.
According to Faucitano et al. (2004), in many cases during the attribution of the mar- bling score to a given sample, the panellists could face with heterogeneous distribution of intramuscular fat.In other words, the fat concentration is present in a certain region and is not distributed throughout the sample, leading to different scores among panellists.However, this problem is mitigated with the use of CVS, since it considers the total muscle area independent of the way intramuscular fat is distributed.

Beef
Similar to pork dataset, the analysis of beef dataset began by searching for the the smallest number of n to be considered in the modelling step in order to obtain an adequate accuracy.Due to the lower number of available samples in the beef dataset with respect to the pork dataset, in this case the maximum value of n was set equal to three.
Regarding each marbling score, Figure 9 shows that, by using only one sample (n = 1) in the modelling step, the median accuracy value was always lower than 50%, with outliers presence in all the scores.Using two samples (n = 2), only marbling score two presents outliers.However, using three samples (n = 3), the accuracy values for score four show a significant increase in terms both of the median and of the average value, as reported in Table 2.

k-NN parameter
The modelling step was performed by varying k to discover the best k -NN parameter value to build a good prediction model.Thus, this step started from k = 1 and increased until reaching a stable accuracy within the limit of available samples.For pork, satisfactory accuracy values were obtained starting from k = 2 and were almost stable from k = 3 to k = 5, as shown in Figure 10: the best performance was obtained from three to five neighbours (3 k 5), as it is also shown in Figure 12, where the average accuracy values are reported.

Other issues
An advantage of the proposed method lies in its ability to efficiently deal with muscles with different aspect in terms of colour and contrast.For example, colour variations among different meat qualities as PSE (Pale, Soft, and Exudative) and DFD (Dark, Firm, and Dry) is automatically normalised before performing marbling evaluation.In Pang et al. (2014) it was necessary to apply a method based on homomorphic filtering to reduce uneven illumination influence and light reflection for beef accurate segmentation.
Other CVSs require to specify many values to properly configure the imaging system, focusing on a single problem scenario and sample-based features to detect marbling.For example, Liu et al. (2012) and Huang et al. (2013) proposed tools for automatic pork marbling detection, while Jackman et al. (2009) and Chen and Qin (2008) proposed a specific algorithm for beef segmentation.Conversely, the proposed approach mitigates the effects of different environmental setups for image acquisition and minimises the number of parameters to be set.

Conclusion
The proposed CVS showed to be a viable alternative compared to traditional assessment of meat marbling, since it is capable to reduce the dependence on human experts and mitigates problems of panellists evaluation by few labelled samples.
Our CVS obtains marbling meat score by an objective and fast assessment, since machines can evaluate multiple images with no pause.This implies also lower costs in comparison to panellists, who need training and require much longer times to perform the same task.
This alternative is suitable to production lines in slaughterhouses, and does not require that the images are acquired within a controlled environment.
Panellists are more susceptible to misclassification due to low marbling levels or variability of fat distribution.The proposed approach performs marbling identification and score prediction in different scenarios (low or high marbling level; dark or pale muscles) based on a ML algorithm.
A variety of research works dealing with similar tasks applied the SVM or the ANN algorithms, but for these algorithms the proper selection of the model parameters is not a trivial task, and commonly is strictly related to the specific problem at hand.Alternatively, looking for a simpler solution, we investigated the use of k-NN and achieved good results for two different muscle foods (pork and beef), also using a limited number of samples during the modelling step with respect to similar approaches already reported in the literature.
In fact, the results reported in the present work demonstrated that the k -NN approach can correctly identify marbling score using few samples of each grade.
Further research work is currently aimed at verifying the device independence of the proposed approach, by using different digital cameras and smartphones in the image acquisition step .

Acknowledgments
The authors would like to thank CAPES (Brazilian Agency) for their financial support.

Figure 3 :
Figure 3: Background removal, keeping the Region of Interest (ROI) only.

Figure 4 :
Figure 4: Example of illumination normalisation in pork sample

Figure 5 :
Figure 5: Contrast enhancement and ROI erosion of pork and meat samples

Figure 6 :
Figure 6: Thresholding and noise removal of pork and beef samples ROI

Figure 7 :
Figure 7: Accuracy of k -NN algorithm for pork prediction models built with increasing number of samples

Figure 8 :
Figure 8: Panelists variation and CVS inside variation range.

Figure 9 :
Figure 9: Accuracy of k -NN algorithm for beef prediction models built with increasing number of samples

Figure 10 :
Figure 10: Boxplot of pork model accuracy obtained for different k values

Figure 12 :
Figure 12: Comparison of near neighbours (k ) evaluated in experiments for pork and beef.

Table 1 :
Average accuracy values (ACC) and standard deviation values (STD) for different numbers of samples (1 n 5) by different marbling scores (1, 2, 3 and 4) for the pork dataset.
Table1reports the average accuracy values of the data shown in Figure7, together with the relevant standard deviations.In general, the smallest accuracy values were always obtained for score three, independent of the number n of samples.Since the accuracy values were determined by comparison with the corresponding assessments made by panellists, this result is not surprising.In fact, in the traditional approaches used for marbling assessment, intermediate scores are those most susceptible to divergence among different assessors, due to subjectivity.Figure8is an example of panellists subjectivity.Figure

Table 2 :
Average accuracy values (ACC) and standard deviation values (STD) for different numbers of samples (1 n 3) by different marbling scores (1, 2, 3 and 4) for the beef dataset.