Edge Databases – from trends to use cases
Data is decentralized. Cloud computing is centralized. Forcing the decentralized world into the centralized cloud topology is not only inefficient, but also economically, ecologically and socially wasteful – and sometimes simply impossible.
To drive digitization and extract value from decentralized data, we need to give the cloud an edge, or more precisely add Edge Computing. Edge computing is a decentralized topology for storing and processing data as close as possible to the data source, i.e., the place where the data is produced, at the edge of the network.
Valuable data is increasingly generated in a decentralized manner – outside traditional and centralized data centers and cloud environments. The dominance of centralized cloud computing approaches slows down digitization and the use of this existing decentralized data. Therefore, according to Gartner (2023) “Edge computing is integral to digital transformation”, and we need infrastructure technologies for the edge that enable developers to quickly and reliably work with decentralized edge data.
Edge Database (Foundation for Edge Data Management) is a new type of database that addresses these requirements. Developers need fast local data persistence and decentralized data flows (Data Sync) to implement edge solutions. Edge Databases solve these core edge functionalities out-of-the-box, allowing application developers to quickly implement edge solutions.
Megatrend to decentralized Edge Computing
By 2030, 30+ billion IoT devices will be creating ~4.6 trillion GB of data per day. The growing numbers of devices and data volume, variety, and velocity, as well as bandwidth infrastructure limitations, make it infeasible to store and process all data in a centralized cloud. On top, new use cases come with new requirements, a centralized cloud infrastructure cannot meet. For example, soft and hard response rate requirements, offline-functionality, and security and data protection regulations.
These trends accelerate the shift away from centralized cloud computing to a decentralized edge computing topology. Edge computing refers to decentralized data processing at the “edge” of the network. For example, in a car, on a machine, on a smartphone, or in a building. Hardware specifications do not capture the definition of an “edge device”. The crucial point is rather the decentralized use of data at, or as close as possible to, the data source.
Edge computing itself is not a technology but a topology, and according to McKinsey, one of the top growing trends in tech in 2021. The technologies needed to implement the edge computing topology are still inadequate. More specifically, there is a gap in basic “core” edge technologies, so-called “software infrastructure”. This gap is one of the main reasons for the failure of edge projects.
Needed: Infrastructure Software for Edge Computing
With computing shifting to the edge of the network, the needs of this decentralized topology become clear:
Need for fast local data storage
→ i.e. a machine on the factory floor collects data on stiffness, friction, pressure points. There is limited space on the device, and typically no connection to the Internet. Even with an Internet connection, high data rates quickly push the available bandwidth, as well as associated networking / cloud costs, to the limit. To be able to use this data, it must be persisted in a structured manner at the edge, e.g. stored locally in a database.
Need for reliable on-device data flows
→ i.e. the car is an edge device consisting of many control units. Therefore, data must be stored on multiple control units. In order to access and use the data within several of the control units of the car, the data must be selectively synchronized between the devices. A centralized structure and thus a single point of failure is unthinkable.
Need for edge-to-edge-to-cloud data flows
→ i.e. in a manufacturing hall: Typically, you will find any number of diverse devices from sensors to brownfield to greenfield devices, and no internet connectivity. At the same time, there are diverse employee devices such as tablets or smartphones, as well as central PCs, and a cloud. To extract value from the data, it must be available in raw, aggregated, or summary form, in different places. This means it needs to be synchronized efficiently and selectively, with possible conflicts resolved.
Need for flexible edge data management
→ e.g. with the rise of IoT, time-series data have become common. However, time series data alone is usually not sufficient, and needs to be combined with other data structures (like objects) to add value. At the same time, a push to standardize data formats in industries (e.g. VSS in automotive or Umati in Industrial IoT) requires that the database supports flexible data structures.
Developing solutions without software infrastructure on an individual level is possible, but has many drawbacks:
Custom in-house implementations are cumbersome, slow, costly, and typically scale poorly. Oftentimes, applications or certain feature sets become unfeasible to deliver because of the lack of core software infrastructure. Legacy code and individual workarounds create problems over the lifetime of a product. Instead of a thriving ecosystem, only a few big players are able to implement edge solutions. Innovation and creativity are limited. An edge database is part of the solution and enables the entire edge ecosystem to build edge applications faster, cheaper and more efficiently.
What is an Edge Database?
An Edge Database is a type of database specifically tailored to the unique requirements of the Edge Computing topology. Edge Databases run directly on-device, locally, and make it easy for app developers to access decentralized data from edge devices when and where needed. Using an Edge Database removes the burden of implementing ways to synchronize data, which is non-trivial, time-consuming, risky, and brings ongoing maintenance needs. Let’s look at this in more detail:
First, an Edge Database is optimized for resource efficiency (CPU, memory, …) and performance on resource-constrained devices (embedded devices, IoT, mobile). It has a small footprint of a few megabytes max. Traditional databases such as MySQL or MongoDB are too large and slow for typical edge devices, making them unsuitable for computing at the edge. Nevertheless, with integrations like the one between ObjectBox and MongoDB, developers can now combine ObjectBox’s on-device efficiency and offline-first capabilities of Edge Databases with MongoDB’s scalable cloud platform to enable seamless, bi-directional synchronization between the edge and the cloud.
An edge device without data flows to/from other devices is just a data island with very limited utility. Accordingly, an Edge Database must support the management of decentralized data flows. There is no more efficient way than at the database level. This ideally includes a range of conflict resolution strategies due to the decentralized and multi-directional structure of the Edge.
Last not least, data security is of growing importance and data in motion needs to be protected. Data at rest is on a database level often protected by the OS and therefore less of a concern for most applications.
When do you need an Edge Database?
Most IoT applications need to store and synchronize data. An Edge Database is always useful when functions / applications are planned that:
- should work offline and independent of an internet connection
- need to guarantee fast response times
- work with a lot of, possibly high-frequency data
- need to serve many devices at the same time
- need historical data
In addition, developers also often decide to use an Edge Database to save time and nerves, or to be able to react quickly and flexibly to future requirements.
Edge Database Use Case Example in Manufacturing
Today, you can find everything from low-frequency brownfield devices to high-frequency greenfield devices on a factory floor. As a rule, the machine controllers in use are not designed to store or transmit data. They usually lack not only the functionality, but also the resources to support this. Therefore, additional edge devices are often needed to collect, analyze and interpret the huge amounts of data that each machine produces on site. For such an edge device, rapid data persistence and ingestion, and efficient data flow from edge-to-edge and edge-to-cloud are at the heart of value creation. The clear separation of machine control and edge data processing unit ensures that there is no risk of unintentional interference with the machine controller. An edge device with a powerful edge database can support multiple use cases on the shop floor today:
1. Operational efficiency
Process optimization along the line to increase quality and reduce damage. When the first machine in a production line uses a new batch of material, i.e. in sheet metal processing, one of the first steps is to cut a sheet to the required size. At this stage, the machine can already detect the differences in the metal compared to a previous batch (deviations are allowed within the DIN standard). With an Edge device this data can be evaluated, and the relevant information passed on to the next machine. With this data machines further down the line can avoid damage / breakpoints of the material.
2. Condition monitoring
Continuous machine condition monitoring reduces downtime and increases maintenance efficiency. A constant stream of high-frequency machine data is compared against the fingerprint of the machine. Any slight deviation is immediately detected and reported. Catching deviations early reduces down-times and costly repairs.
3. Historical Data
Historical data is stored for learning and training to optimize the production line. With an Edge Database, the data is persisted and thus available in the event of faulty behavior. In case of an error, the data preceding the incident can be analyzed and used to find the causes and predict, or even avoid, such an error in the future. Chances are that “fuzzy expert knowledge” already available at the production site can be translated into deterministic rules when tested with these data sets.
The future of Edge Databases
Edge computing provides numerous benefits and enables many applications and functionalities that are only possible with edge computing. However, only a few (usually large) players have been able to create value in edge computing projects, gaining competitive advantages. One reason is a lack of basic edge software. A thriving edge ecosystem necessitates edge software infrastructure that addresses the fundamental recurring needs of edge projects. Edge databases are a critical component in the development of such an ecosystem.
Looking ahead, the emergence of on-device vector databases, coupled with small language models (SLMs), is transforming the landscape of AI applications. These technologies enable AI apps to run directly on edge devices, providing long-term memory, improving performance, and significantly reducing resource consumption. By processing data locally, they eliminate the need for constant cloud connectivity, enhancing privacy and efficiency. Companies like Apple have already embraced on-device AI (Apple Intelligence), showcasing its potential to deliver advanced functionalities seamlessly. This shift represents a game-changer, making AI more sustainable, scalable, and integrated into everyday use.