NOTAS BREVES

The future of remote sensing: harnessing the data revolution

El futuro del sensoramiento remoto: aprovechamiento de la revolución de los datos

 

Charles Toth¹

¹ Department of Civil, Environmental and Geodetic Engineering, The Ohio State University, Columbus, OH, USA Email: toth.2@osu.edu

 

---

ABSTRACT

Geospatial data has been used increasingly in applications far beyond the traditional mapping and engineering fields. Location-based services and location-aware data have been transforming every aspect of our lives. Both augmented reality (AR) and virtual reality (VR) systems are based on spatial data and spatial data processing, and can now analyze an individual’s field of view to provide information, such as the identification of items and their properties or recommendations of actions, and then 3D visualizations in time of location-attributed datasets, respectively. The source of data is an ever-growing set of sensors and data collection devices, each providing data with specific sensor-determined spatial, spectral and temporal resolution, and error properties. Besides mapping and Earth sciences, applications abound, ranging from autonomous vehicles to smart city implementations, human/machine collaborative defense operations and surveillance, etc. The growing amount of geospatial data, processed by new big data methods, are already driving developments and will create new applications and industries that will have major societal impact over the next decade and beyond.

RESUMEN

Los datos geoespaciales han cobrado un gran auge en diversas aplicaciones que distan de los campos tradicionales de la cartografía e ingeniería. Actualmente, los servicios basados en localización permiten detectar una ubicación actual para luego utilizar estos datos como fuente de información, lo que ha transformado cada aspecto de nuestras vidas. Los sistemas de realidad virtual y aumentada, respectivamente, se basan en la utilización de datos espaciales y sus procesamientos, donde actualmente es posible analizar el campo de visión de un individuo para proporcionar información como ser la identificación de elementos, sus propiedades o recomendaciones de posibles acciones, como así también visualizaciones en 3D del conjunto de datos atribuidos a la ubicación. Las fuentes originarias de datos están en continuo crecimiento, y cada vez son mayor la cantidad de sensores y dispositivos de recopilación de datos integrados, donde cada uno proporciona propiedades específicas de resolución espacial, espectral y temporal y sus errores asociados. Además de aplicaciones en cartografía y en Ciencias de la Tierra, otros campos han sido explorados como el de vehículos autónomos, implementaciones de ciudades inteligentes, fabricación colaborativa entre personas y maquinas, gestión ambiental y de desastres naturales, funciones de asistencia médica y geriátrica, operaciones de defensa y vigilancia, etc. El creciente volumen de datos geoespaciales procesados mediante los métodos llamados de “big data” han impulsado nuevos desarrollos, aplicaciones e industrias, que generarán un gran impacto en la sociedad durante la próxima década e incluso más allá.

---

 

REMOTE SENSING

The field of Remote Sensing has been vastly expended recently, and its applications have far surpassed the historical limits of mapping, based on optical image acquisition from satellite and airborne platforms. By now, Remote Sensing encompasses any noncontact techniques to observe and map some of the physical parameters of our environment, covering the electromagnetic spectrum from radio to the optical band, and including the sensing of the Earth’s magnetic and gravitational fields. Driven by technological developments, sensors have become ubiquitous and can be found in almost every facet of our lives. In particular, imaging sensors have shown dramatic improvements in performance and variety, offering a broad range of spatial and temporal resolutions with spectral capabilities from pan to multispectral to hyperspectral bands. Technological developments have also allowed for a proliferation of remote sensing platforms, which are now available from space to air to ground to subsurface. Figure 1 illustrates the diversity and the range of the most widely used sensors and platforms.

Figure 1. Remote sensing platforms and sensors.
Figura 1. Plataformas de sensoramiento remoto y sensores.

Concerning the trends in remote sensing, the developments are very strong in the less expensive, consumer-grade sensors and small, mobile platforms. For example, autonomous and assisted vehicle technologies, one of the hottest fields currently, require simple yet reliable sensing capabilities. In particular, research and innovation on laser sensing are an extremely hot area, as point clouds offer a true-scale direct 3D representation of the vehicle environment at day and night. In comparison, optical sensing has become extremely advanced, mainly due to widespread smartphone applications. In general, achieving better mobility and autonomy drives the platform developments, primarily in the consumer market. UAS systems have shown remarkable progress in terms of performance, reliability, increasing capabilities, and ease of use. Driverless vehicles are on the verge of being introduced to normal use. In both cases, the enabling technology is remote sensing, combined with advanced data science methods. Figure 2 shows a point cloud obtained from a mobile platform, aimed at supporting driverless vehicle technologies by providing a high-definition 3D map of the environment that provides accurate vehicle positioning when GPS signals are not available or the pavement markings on the road are not visible. Note that the point cloud is a base layer, based on the object space is segmented and object features are extracted and stored in vector and other metadata formats that can effectively support high-level search and match operations.

Figure 2. High-definition point cloud of a section of the OSU Campus, acquired from a mobile platform.
Figura 2. Nube de puntos de alta definición de una sección del Campus OSU, adquirida desde una plataforma móvil.

SENSOR INTEGRATION

Sensors, both at the professional and consumer levels, are hardly used alone on any current platform, and multi-sensor data acquisition has become the norm of practice in remote sensing. For example, an airborne mapping system usually includes a GPS/IMU sensor combo for accurate sensor georeferencing and then one or more optical sensors, such as digital cameras and/or LiDAR sensors. On the consumer level, smartphones typically sport a large number of sensors, including GPS, IMU, magnetometer, cameras, pressure, brightness, and sound sensors, etc., in a remarkably small enclosure, creating a powerful navigating and mapping device. Sensor integration represents the methods to combine data streams coming from complementary and, many times, redundant sensor measurements. Before performing any sensor integration, however, the sensors and their spatial and temporal relationship should be calibrated to achieve optimal sensor performance. The most essential, low-level process is the georeferencing of the image data, typically accomplished by using the GPS/IMU integrated navigation solution to estimate position and attitude parameters for the imaging sensor at the time when the image data is acquired. Augmented reality (AR) and virtual reality (VR) systems, used in consumer or professional applications for visualization, require accurate sensor and viewpoint georeferencing; Figure 3 shows a smartphone app with AR, the Pokemon Go. An example for higher-level sensor integration is the fusion of optical imagery and LiDAR data, where accurate sensor georeferencing is an essential prerequisite.

Figure 3. Pokemon Go VR app.
Figura 3. App Pokemon Go VR.

The sensor integration of navigation and imaging sensors was the enabling technology that resulted in the unprecedented proliferation of location-based services and the recent, widespread use of locationaware data. In fact, these developments triggered a paradigm shift in the navigation and mapping fields, which, historically, were fairly disconnected. The outstanding, outdoor, anywhere-and-anytime navigation performance of GPS based devices created a tremendous demand for the same level of navigation accuracy in GPS-challenged areas, such as urban areas, in canyons and indoors. While no sensor can replace GPS in that environment, sensor integration has been the only viable solution used currently to approach GPS-level accuracy in certain cases. In particular, imaging sensors are used successfully for positioning, using various computer vision techniques, such as Visual Odometry or Structure from Motion based SLAM (Simultaneous Localization and Mapping) methods. On the mapping or geospatial data acquisition side, there is no contemporary data capture system that would not include some navigation sensor or georeferencing technologies. Professional mapping systems are typically supported by cm-level accuracy, while less expensive, consumer-grade devices generally provide few meter level positioning accuracy.

CROWDSENSING/CROWDSOURCING

Historically, geospatial data, primarily distributed in map formats were acquired by governments and private companies. With the proliferation of inexpensive imaging sensors, the situation has started to change, as consumers are increasingly acquiring large amounts of image data that contain valuable geospatial information. In particular, this is the situation in urban areas and indoors, where people both spend most of their time and use their smart devices. The term crowdsensing is similar to crowdsourcing and they are frequently used interchangeably in the geospatial context; note that the crowdsourcing is about 10 years old, and was originally created by the IT industry. In the developed world, the human environment is getting peppered with inexpensive sensors, deployed on vehicles, infrastructure, humans (wearable sensors), smart devices, etc., and, consequently, consumer sensor data are becoming ubiquitous. Voluntarily or involuntarily, crowdsensing creates an increasingly large volume of data, which has a tremendous potential to extract geospatial information. Note that crowdsensing always provides georeferencing, which is only moderately accurate, yet it is sufficient to organize the sensor data and can be usually refined during subsequent photogrammetric processing. The real question is more about accessing the crowdsensed sensory data, which directly connects to communication and the general IT infrastructure; an area that is rapidly improving in both coverage and speed. The data acquired worldwide, per day, are rapidly increasing and currently estimated to be 50 quintillions (10¹⁸) bytes, where video and image data represent the largest component. Most of the crowdsensed data get archived, and the amount of digital data stored worldwide is sharply growing. Obviously, crowdsensing data are currently no match for high quality geospatial data, acquired by metric quality sensors, yet as technology and data processing techniques continue to advance, there will be a point when crowdsensed data will provide comparable quality geospatial information to present-day, state-of-the-art mapping technologies. Moreover, in terms of observing the object space, it is clear that crowdsensing data will surpass the operational envelop of traditional remote sensing. By definition, crowdsensing can only observe the object space where the “crowd” is, which are usually areas of high interest, such as densely built up areas or transportation corridors, etc. Estimates vary on a wide scale, but experts generally agree that the number of sensors that qualify for the crowdsensing status, will be, at least, an order larger than the population of the Earth. The ultimate question is how to exploit the potential of crowdsensing data. Emerging methods, including Data Analytics and Big Data are the likely answer to it. Figure 4 shows two heat maps, acquired by people exercising in the Columbus, OH area, using the Strava app; note the difference between the running and biking routes, and spatial distribution of the routes in general, as they correlate to the general quality of living in the various neighborhoods.

Figure 4. Heat map of people exercising (running –left, bycicling –right) in the Columbus, OH area, acquired by Strava.
Figura 4. Mapa de calor de gente ejercitándose (corriendo –izquierda, bicicleta –derecha) en el área de Columbus, OH, USA, adquirido mediante Strava.

DATA REVOLUTION

Data Science has existed since the beginning of the digital age, but neither the applications nor the computing technology were ready to harness the potential of this discipline. Due to the explosive developments in sensing, geospatial sensors started to produce an ever-increasing amount of data recently, and soon Big Data have become a reality. On the computing front, cloud computing has gained dominance and, for the first time, nearly unlimited processing and storage capacity are offered at an affordable price. Geospatial data is one of the big beneficiaries of the ongoing developments of the new era in research and application known as Data Science, Data Analytics, or Big Data. Data streams at petabyte (10¹⁵ bytes) and exabyte (10¹⁸ bytes) scales are becoming common and widely available via warehouse-scale computers and the internet. Figure 5 shows the trend of information creation worldwide. In addition, due to cloud computing, very large computers are also widely available, providing the base for massive processing. It is important to emphasize that besides the highly-automated information processing, the real potential of harnessing data science is the capability to extract additional information that has not been feasible in the past. For example, using satellite data to monitor changes in a region where some health epidemic occurred, the correlation between the spread of the disease and the state of vegetation can be established. Besides classical mapping, Big Geo Data are expected to contribute to the society in new capacities by offering massive datasets on human behavior containing information relevant to understanding the big problems facing society and individuals today.

Figure 5. Increasing pace of data creation worldwide.
Figura 5. Incremento en la creación de datos en el Mundo.

FINAL REMARKS

The geospatial science and engineering field is experiencing rapid growth, and is expected to advance significantly in the coming years. The developments are primarily fueled by the enormous amount of geospatial data, acquired by ever-powerful sensors, both professionally and crowdsensed, and the availability of big data methods. A key question is how geospatial data can harness the ongoing, data science revolution, as it has become clear that traditional processing cannot keep up with the everincreasing volume of data coming from satellites, UAS, driverless vehicles, smart devices, etc. Exploiting the potential of crowdsensed data is a likely, key element of this process, as it provides for a sustained observation of our environment, offering both currency and accuracy.

BIO

Charles K. Toth is a Research Professor in the Department of Civil, Environmental and Geodetic Engineering, The Ohio State University. He received a M.Sc. in Electrical Engineering and a Ph.D. in Electrical Engineering and Geo-Information Sciences from the Technical University of Budapest, Hungary. His research interest and expertise cover broad areas of spatial information sciences and systems, including photogrammetry, multi-sensor geospatial data acquisition systems, LiDAR, high-resolution imaging, surface extraction, modeling, integrating and calibrating multi-sensor systems, georeferencing and navigation, 2D/3D signal processing, and mobile mapping technologies. He has published over 350 peer-reviewed journal and proceedings papers, and is the recipient of numerous awards, including the 2009 APSRS Photogrammetric Award, United States Geospatial Intelligence Foundation (USGIF) Academic Achievement Award, 2015, ISPRS Schwidefsky Medal Winner 2016, several Lumley Research Awards from Ohio State, and various best papers awards.
Known worldwide for his visionary contributions to mobile mapping, Professor Toth was a key architect of the concept development, and then did significant research in sensor georeferencing and digital imaging technologies. He is generally credited with the introduction of the term “direct and indirect georeferencing” in the photogrammetric community. As one of the founding fathers of the Mobile Mapping Technology symposium series, Dr. Toth has actively contributed to all of them. He is very devoted to education and mentoring of the next generation of photogrammetric professionals, and has been a major contributor to the annual Summer Schools on mobile mapping. Widely recognized in the international mapping community, he has held many senior leadership positions in national and international societies, and was the President of the American Society of Photogrammetry and Remote Sensing, served as the ISPRS Technical Commission I President for 2012-2016, and is the ISPRS 2nd Vice-President for the period of 2016-2020.