A Drone-based Application for Scouting Halyomorpha halys Bugs in Orchards with Multifunctional Nets

In this work, we consider the problem of using a drone to collect information within orchards in order to scout insect pests, i.e., the stink bug Halyomorpha halys. An orchard can be modeled as an aisle-graph, which is a regular and constrained data structure formed by consecutive aisles where trees are arranged in a straight line. For monitoring the presence of bugs, a drone flies close to the trees and takes videos and/or pictures that will be analyzed offline. As the drone’s energy is limited, only a subset of locations in the orchard can be visited with a fully charged battery. Those places that are most likely to be infested should be selected to promptly detect the pest. We implemented the proposed approach on a DJI drone and evaluated its performance in the real-world environment.


I. INTRODUCTION AND MOTIVATION
Nowadays, autonomous vehicles are extensively used in orchards [1]- [4], and field monitoring of parasites is a fundamental operation in crop management. to improve crop pest management, Unmanned Aerial Vehicles (UAVs) can be used as an innovative technology enabling fast, efficient, and real-time monitoring of insect infestations. This study is presented in the context of Haly.ID project [5], which aims at implementing a prototype of a digital platform for monitoring the presence of the brown marmorated stink bug Halyomorpha halys (HH) in crop fields, which represents a high-invasive pest of global importance for many crops (Fig. 1). In fact, the HH seriously damages fruit production in southern Europe (pears in Italy in particular), and United States. Due to the above considerations, we develop an application for planning and tracking the tasks of a drone that is in charge of scouting the HH inside an orchard. The components of our system are as follows: (i) a smartphone, which acts as a controller for the UAV, which initially receives input from the user and then determines a route for the UAV, and (ii) a UAV, which gets commands from the smartphone and then follows the route.
We make the following contributions: i) we propose a new data structure to represent orchards/vineyards; ii) we design an intuitive Android-based application that allows the user to select different PoIs in the orchard to be monitored and outputs a route for the drone based on the chosen path planning This work was partially supported by the "GNCS -INdAM", by "HALY-ID" project funded by the European Union's Horizon 2020 under grant agreement ICT-AGRI-FOOD no. 862665, no. 862671, and by MIPAAF. algorithm; iii) we test the prototype system using a DJI drone equipped with a Zenmuse H20 camera.

A. Background
In this work, we consider an application that requires a drone inside an orchard, with a particular focus on the timely detection of bugs on trees. Orchards (and vineyards) are formed by several consecutive aisles, which in turn contain many trees along a straight line. Modern orchards use multifunctional nets on top and on sides not only to protect the crops from hail (see Fig. 2), but also to make it more difficult for for pests (both insects and birds) to enter and infest the trees [6], [7]. Due to the nets, drones cannot fly over the trees to take pictures of bugs, but can only move along the aisles. For detecting bugs in modern orchards, a drone, moving along aisles, is more effective than a ground or handheld device because it can cover the whole area in a shorter time, and no special pre-processing on the ground is required. Especially, by simply changing its height, a drone can take video/photos at the bottom or top of each tree, possibly with the same focal distance, thus preserving the same spatial resolution. However, due to battery limitations, drones may not be able to cover the entire orchard. The drone's energy consumption does not depend only on the orchard size, but also on the payload (e.g., an RGB sensor is lighter than multi-or hyper-spectral sensors, but with limited spectral information) and on the speed. When recording video or taking photos, the drone must fly slowly (or even stop, i.e., hovering) to preserve adequate camera stabilization. Limited speeds imply a longer coverage time and therefore, depending on the characteristics of the drone, a higher energy consumption.
Given that the drone has limited energy for flying and image/video recording, we assume that the observable positions on the trees can be prioritized. Specifically, we associate a priority, called reward, to each observable position. The reward can be decided by analyzing the historical time series of bug numbers detected in each PoI, along with weather conditions that have been recorded in past years, and also by considering entomological knowledge. So, our objective is to plan a drone's route in the orchard for taking pictures or videos (to scout bugs) on the most profitable locations (with larger reward) under a given limited drone's battery.

B. System Overview
We implemented our proposed solution as follows. The UAV is a powerful DJI Matrice 300 RTK [8] which allows developers to design custom applications by using the DJI Mobile SDK (MSDK). The MSDK provides interfaces for the flight and camera control of the UAV. We use a Redmi Note 7 smartphone with Android 10 as a system server. In the following, we first describe our new data structure which we use to represent the orchards on which we can then build missions/routes for the drone. We then describe in much more detail our developed application for the drone.

C. Data Structure
To properly model the orchards in our application, we devised a graph-based data structure simply called orchard. This data structure takes in input three parameters: the number of rows (aisles) m, the number of columns n, and the number of possible observable positions on each tree l. On each row there are n trees, and therefore the orchard has exactly mn trees and mnl different positions to be observed by the drone. Then, we can create a graph O(m, n, l) = (V, E) formed by a set of vertices V and a set of edges E. The set V is composed by the actual set of tree roots of the orchard (brown vertices in Fig. 3), and the set of observable positions by the drone (blue vertices in Fig. 3). The set E is composed by the extra-tree connections, i.e., edges between trees in a same aisle, and the intra-tree connections, i.e., edges between observable positions in a tree. We associate to each observable point a key value and an auxiliary data structure, e.g., a list. In particular, we will use the key to tag and uniquely identify the data collected during the drone's mission, while the auxiliary data structure to store them. Those properties allow us to subsequently manipulate the harvested media by the drone preserving their spatial location in the orchard in an offline way. So, given a key value, we can retrieve the relative marked tree and vice versa. Fig. 3 depicts an example of an orchard O(3, 4, 3) with 3 rows (aisles), 4 columns, and 3 possible observable positions for each tree (having so 12 trees and 36 positions to be observed). It is worthy to point out that the drone cannot cross the aisles because of the tree's branches and foliage, and cannot fly over because of the net. To complete our data structure we need to associate a cost to each edge to represent the energy cost for the drone. Indeed, a drone is an energy-constrained flying vehicle that has a given battery capacity, and when the drone moves flying along edges, it spends energy from its battery. Finally, we need to associate a reward with each vertex that characterizes the importance of doing a specific task on vertices. The reward provides meaningful values only for the set of observable positions while it provides zero values for the set of tree roots. The reward is assigned to the drone only the first time the drone performs tasks on that particular vertex. Furthermore, the reward for that vertex is no anymore obtainable. However, if the drone repeatedly travels on the same edge multiple times, the energy cost is considered multiple times.

D. System Design
The proposed application comprises of the following phases: 1) The orchard inspection phase: In this phase, we perform an analysis concerning the orchard, functional to the subsequent phase. The drone flies over the net starting from the home, which is usually at the beginning of an aisle, until it reaches the opposite end of the aisle, then moves to the next aisle and again locates its two extremes. In our current experimental version, we rely on the drone's real-time kinematic (RTK) system for retrieving the global coordinates with an accuracy of less than 10 cm. During the pre-processing, we will also localize the Internet of Things (IoT) devices (a small trap-camera, a certain number of meteorological sensors) placed in the orchard [9]. From this information, we model the orchard as a graph. Knowing that in modern orchards trees are placed 3 m apart from each other along an aisle, we can build the graph O(m, n, l) used by the drone, identifying also some special positions where the IoT devices are.
2) The route planning phase: In this phase, we generate the drone's route from the home point. In this demo, we consider a mission that consists of a single aisle. Fig. 4. Screenshots of the prototype application.
Just as an example, Fig. 4 shows two screenshots taken from the smartphone. In Fig. 4 (left), the application allows to specify the start and end positions of the mission collected in the previous phase, along with the drone's altitude and speed. The drone's mission follows a set of fixed WayPoints (WPs) along the straight line that connects the start and endpoints. The WP distance is set to 3 m. Note that there is no risk the drone stops in front of an empty space since the tree branches form a continuous wall. Finally, concerning the actions to perform at each WP, the user can specify the drone's rotation angle (yaw), the camera rotation (gimbal pitch), and the focal length of the camera. This latter parameter is of particular importance because it drives the camera focus. In Fig. 4 (right), it is shown the generated mission (WPs in blue) along with the custom parameters chosen by the user, and the RTK pos err, that is the accuracy of the GPS when using the RTK system.
3) The drone scouting phase: In this phase, the drone performs the mission after the user presses the Start button shown in Fig. 4. The drone executes its mission flying through the WPs (i.e., PoIs). The drone stops at the PoIs for performing the sequence of planned actions. For each PoI, either several photos can be taken hovering at the same altitude or the drone can take photos at different heights of the tree (bottom, center, top). In the former case, since the distance between the target (tree, pear, bug) and the lens change, the focal length has to be changed for each photo. Cropping of the images might be required to let all the photos overlap the same. In the latter case, instead, the focal length is unchanged. This latter case is more suitable if an orthomosaic software will be adopted to rebuild the digital image of the orchard. In this demo, we just take ℓ photos at each PoI at the same altitude, and we postpone different implementations to subsequent versions of our application. As a buy product of the drone scouting phase, the drone might, under some circumstances, collect the data from the IoT devices. Once the drone finishes the mission, it lands at the home point, and an offline analysis for detecting the HH by using ML techniques is started. Also in the future, we plan to elaborate the images online, using a Raspberry Pi housed on the top of the drone.
4) The data analysis phase: In this phase, we perform an analysis of the harvested data to detect and estimate the presence of bugs inside the orchard. Using an appropriate prebuilt classification model, we first infer the presence of HH on the trees. Secondly, we try to estimate from the same data the numerousness of the HH population in the orchard. This phase is only implemented offline, for the time being. Notice that, the higher is the accuracy of our models and the timeliness of the response, the higher the quality of our in-field detection and mitigation system will be.

III. DEMO INFORMATION
This experiment has been done in an orchard in Italy with 12 rows and 15 columns, and setting 3 observable positions, i.e., an O(12, 15, 3). The DJI Matrice 300 RTK (see Fig. 1) is able to fly for up to 55 min having 2 batteries with capacity of ≈ 6000 mAh. As said before, the drone just take pictures/video on the fly, and the actual detection of the HH is done offline.

IV. DISCUSSION AND OUTLOOK
Drones are revolutionizing agriculture by offering farmers efficacy and efficiency, and early pest scouting is a major application of drones in this area. The impact of drones is constrained by the energy required for flying and collecting data. Thus, an important step towards bug detection is to plan an efficient route for the drone in the orchard by selecting the areas/trees in which to capture images or videos. In the immediate future, we need to improve pictures quality to effectively train ML algorithms.