Why do we need Edge Computing for a sustainable future?

Why do we need Edge Computing for a sustainable future?

Centralized data centers use a lot of energy and water, emit a lot of CO2, and generate a lot of electronic waste. In fact, cloud data centers are already responsible for around 300 Mt CO2-eq of greenhouse gas emissions [1]. And the energy consumption of data centers is increasing at an exponential rate [2].

While more data centers are switching to green energy [3], this approach is not nearly enough to solve the problem. A more sustainable approach is to reduce unnecessary cloud traffic, central computation, and storage as much as possible by shifting computation to the edge. In our experience, just reducing data overhead and unnecessary data traversals can easily cut 60-90% of data traffic and thus significantly impact the CO2 footprint of an application, as well as costs.

Edge Computing stores and uses data on or near the device on which it was created. This reduces the amount of traffic sent to the cloud and, on a large scale, has a significant impact on energy consumption and carbon emissions.

Why do Digitization projects need to think about sustainability now?

Given the gravity of the climate crisis, every industry needs to assess its potential environmental impact and find ways to reduce its carbon footprint. The digital world, and its most valuable commodity, data, should not be any different. The digital transformation is ongoing and with it electronic devices and IT usage numbers are exploding. Thus, new apps must consider their carbon footprint throughout their lifecycle, especially resource use in operation and at scale [4]. 

Also, think about this: The share of global electricity used by data centers is already estimated to be around 1-3% [1] and data centers generate 2% of worldwide CO2 emissions (on par with the aviation industry) [5]. 54% of all emissions due to cloud data centers are caused by the big hyperscalers (Google, Amazon, Microsoft, Alibaba Cloud) [6]. On top of this, providing and maintaining cloud infrastructure (manufacturing, shipping of hardware, buildings and lines) also consumes a huge amount of greenhouse gasses [7] and produces a lot of abnormal waste (e.g. toxic coolants) at the end of life [8].

sustainable edge computing

Bearing that in mind, the growth rate for data center demand is concerning. The steady increase in data processing, storage, and traffic in the future, comes with a forecasted electricity consumption by data centers to grow by 10% a year [9]. In fact, estimations expect the communications industry to use 20% of all the world’s electricity by 2025 [10].

sustainable edge computing

Shifting to green energy is a good step. However, a more effective and ultimately longer term solution requires looking at the current model of data storage, filtering, processing and transferal. By implementing Edge Computing, we can reduce the amount of useless and wasteful data traversing to and from the cloud as much as possible, thus reducing overall energy requirements in the long term. Of course, everyone can make a difference with their daily behavior and for developers that is especially true: Applying green coding principles helps producing applications that produce lower CO2 emissions over the whole app lifetime. 

What is Edge Computing?

Until recently 90% of enterprise data was sent to the cloud, but this is changing rapidly. In fact, this number is dropping to only 25% by 2025, according to Gartner. By then, most of the data will be stored and used locally, on the device it was created on, e.g. on smartphones, cars, trains, machines, watches. This is Edge Computing, and it is an inherently decentralized computing paradigm (as opposed to the centralized cloud computing approach). Accordingly, every edge device needs the same technology stack (just in a much smaller format) as a cloud server. This means: An operating system, a data storage / persistence layer (database), a networking layer, security functionalities etc. that run efficiently on restricted hardware.

As you can only use the devices’ resources, which can be pretty limited, inefficient applications can push a device to its limits, leading to slow response rates, crashes, and battery drain.

edge device architecture

EDGE DEVICE ARCHITECTURE

Edge Computing is much more than some simple data pre-processing, which takes advantage of only a small portion of the computing that is possible on the edge. An Edge Database is a prerequisite for meaningful Edge Computing. With an Edge Database, data can be stored and processed on the devices directly (the so-called edge). Only useful data is sent to the server and saved there, reducing the networking traffic and computing power used in data centers tremendously, while also making use of the computing resources of devices which are already in use. This greatly reduces bandwidth and energy required by data centers. On top, Edge Computing also provides the flexibility to operate independently from an Internet connection, enables fast real time response rates, and cuts cloud costs.

Why is Edge Computing sustainable?

Edge Computing reduces network traffic and data center usage

With Edge Computing the amount of data traversing the network can be reduced greatly, freeing up bandwidth. Bandwidth is a measure of the quantity / size of data a network can transfer in a given time frame. Bandwidth is shared among users. Accordingly, the more data is supposed to be sent via the network at a given moment, the slower the network speed. Data on the edge is also much more likely to be useful and indeed used on the edge, in context of its environment. Instead of constantly sending data strems to the cloud, it therefore makes sense to work with the data on the edge and only send that data to the cloud that really is of use there (e.g. results, aggregated data etc.).

Edge computing is optimized for efficiency

Edge “data centers” are typically more efficient than cloud data centers. As described above, resources on edge devices are restricted. Therefore, and as opposed to cloud infrastructure, edge devices do not scale horizontally. That is one reason why every piece of the edge tech stack is – typically and ideally – highly optimized for resource efficiency. Any computing done more efficiently helps reduce energy consumption. Taking into account the huge number of devices already deployed , the worldwide impact of reducing resource use for the same operations is significant.

Edge Computing uses available hardware

There is a realm of edge devices already deployed that is currently underused. Many existing devices are capable of data persistence, and some even for fairly complex computing. When these devices – instead – send all of their data to the cloud, an opportunity is lost. Edge Computing enables companies to use existing hardware and infrastructure (retrofitting),  taking advantage of the available computing power. If these devices continue to be underused, we will need to build bigger and bigger central data centers, simultaneously burdening existing network infrastructure and reducing bandwidth for senselessly sending everything to the cloud.

Cloud versus Edge: an Example

Today, many projects are built based on cloud computing. Especially in first prototypes or pilots, cloud computing offers an easy and fast start. However, with scale, cloud computing often becomes too slow, expensive, and unreliable. In a typical cloud setup, data is gathered on edge devices and forwarded to the cloud for computation and storage. Often a computed result is sent back. In this design, the edge devices are dumb devices that are dependent upon a working internet connection and a working cloud server; they do not have any intelligence or logic of their own. In a smart home cloud example, data would be sent from devices in the home, e.g. a thermostat, the door, the TV etc. to the cloud, where it is saved and used.

Cloud vs Edge

If the user would want to make changes via a cloud-based mobile app when in the house, the changes would be sent to the cloud, changed there and then from there be sent to the devices. When the Internet connection is down or the server is not working, the application will not work.

With Edge Computing, data stays where it is produced, used and where it belongs – without traversing the network unnecessarily. This way, cloud infrastructure needs are reduced in three ways: Firstly, less network traffic, secondly, less central storage and thirdly less computational power. Rather, edge computing makes use of all the capable hardware already deployed in the world. E.g. in a smart home, all the data could stay within the house and be used on site. Only the small part of the data truly needed accessible from anywhere would be synchronized to the cloud.

Cloud vs Edge

Take for example a thermostat in such a home setting: it might produce 1000s of temperature data points per minute. However, minimal changes typically do not matter and data updates aren’t necessary every millisecond. On top, you really do not need all this data in the cloud and accessible from anywhere.

With Edge Computing, this data can stay on the edge and be used within the smart home as needed. Edge Computing enables the smart home to work fast, efficiently, and autonomous from a working internet connection. In addition, the smart home owner can keep the private data to him/herself and is less vulnerable to hacker attacks. 

How does ObjectBox make Edge Computing even more sustainable?

ObjectBox improves the sustainability of Edge Computing with high performance and efficiency: our 10X speed advantage translates into less use of CPU and battery / electricity. With ObjectBox, devices compute 10 times as much data with equivalent power. Due to the small size and efficiency, ObjectBox runs on restricted devices allowing application developers to utilize existing hardware longer and/or to do more instead of existing infrastructure / hardware.

Alongside the performance and size advantages, ObjectBox’ Sync solution takes care of making data available where needed when needed. It allows synchronization in an offline setting and / or to the cloud. Based on efficient syncing principles, ObjectBox Sync aims to reduce unnecessary data traffic as much as possible and is therefore perfectly suited for efficient, useful, and sustainable Edge Computing. Even when syncing the same amount of data, ObjectBox Sync reduces the bandwidth needed and thus cloud networking usage, which incidentally reduces cloud costs.

ObjectBox’ Time Series feature, provides users an intuitive dashboard to see patterns behind the data, further helping users to track thousands of data points/second in real-time.

How Edge Computing enables new use cases that help make the world more sustainable

As mentioned above, there are a variety of IoT applications that help reduce waste of all kinds. These applications can have a huge impact on creating a more sustainable world, assuming the applications themselves are sustainable. Three powerful examples to demonstrate the huge impact IoT applications can have on the world:

food-icon

Reducing Food Waste

From farm to kitchen, IoT applications can help to reduce food waste across the food chain. Sensors used to monitor the cold chain, from field to supermarket, can ensure that food maintains a certain temperature, thus guaranteeing that products remain food safe and fresh longer, reducing food waste. In addition, local storage can be used to power apps that fight household waste (you can learn how to build a food sharing app yourself in Flutter with this tutorial).

light bulb

Smart City Lighting

Smart City Lighting: Chicago has implemented a system which allows them to save approx. 10 million USD / year and London estimates it can save up to 70% of current electricity use and costs as well as maintenance costs through smart public lighting systems [10].

water-drop

Reducing Water Waste

Many homes and commercial building landscapes are still watered manually or on a set schedule. This is an inexact method of watering, which does not take into account weather, soil moistness, or the water levels needed by the plant. Using smart IoT water management solutions, landscape irrigation can be reduced, saving water and improving landscape health.

These positive effects are all the more powerful when the applications themselves are sustainable.

Sustainable digitization needs an edge

The benefits of cloud computing are broad and powerful, however there are costs to this technology. A combination of green data centers and Edge Computing helps to resolve these often unseen costs. With Edge Computing we can reduce the unnecessary use of bandwidth and server capacity (which comes down to infrastructure, electricity and physical space) while simultaneously taking advantage of underused device resources. Also with AI growing in popularity, Edge Computing will become very relevant for sustainable AI applications. AI applications are very resource intensive and Edge AI will help to distribute workloads in a resourceful manner, lowering the resource-use. One example of this is an efficient local vector database. ObjectBox amplifies these benefits, with high performance on small devices and efficient data synchronization – making edge computing an even more sustainable solution.

Cross platform Data Sync: a simple example

Cross platform Data Sync: a simple example

Cross platform data sync can be simple: In this tutorial we will show you how you can easily sync data across devices.

Built for fast and effortless data access on and across embedded devices from Mobile to IoT, ObjectBox keeps data in sync between devices for you. The Database and Data Snyc works across platforms (iOS, Android, Linux, Rasbian, Windows, MacOS) and supports a variety of languages with easy native APIs (Swift, Java, Kotlin, C / C++, Flutter / Dart, Golang).

For example, you can sync between an Industrial IoT sensor app in Go and a C++ monitoring application – and a mobile Android app written in Kotlin or Java – and of course an iOS app written in Swift – and… you get the drift 😉

ObjectBox is a high-performance embedded database for Edge Computing with integrated Data Sync. The ObjectBox database is quick to set up and free and easy to use. Our powerful and intuitive APIs are a great match for multiplatform development environments.

Syncing data across devices – a task-list app example

In this tutorial, we are going to sync data across three instances of an example task-list app (written in C++, Go and Java).

With the task-list app, users can create simple text-based tasks and mark them as done. It stores tasks together with their creation dates. There is also a parameter to store the date when the task was completed. It acts as a filter for users to only see unfinished tasks. 

This app is a standard cross platform ObjectBox example that is available for all language bindings. Here are the repositories of each example app that we will be looking at today:

Overview of the example code 

In this section, we’ll quickly review how the the task-list example app uses ObjectBox Sync. For a more detailed description, check out the Sync docs. If you want to see how each of these steps were incorporated into the example code, go to the next section.

Note: The basic use of the database and its sync features is the same for all programming languages. If you haven’t used the ObjectBox DB yet, please refer to the corresponding documentation: C/C++ Docs, Java/Kotlin/Dart Docs, Go Docs, Swift Docs.

For sync to work in any app, we generally only need four things:

  1. The sync-enabled library — this is not the same as the general ObjectBox library and has to be downloaded separately.
  2. Database objects enabled for sync — for this we need include the sync annotation in the ObjectBox schema file.
  3. ObjectBox Sync Server — please apply for a free Sync Trial here to get your own copy of the Sync Server (available for Linux and Docker). Note that this will only have to be started once and in this tutorial we’ll show you how to run the server on Linux. If you are using Docker, follow the steps outlined here.
  4. Start a Sync Client in the app — as one can see from the Sync Client docs, creating and starting a sync client is just a matter of a couple of lines of code.

Important: When syncing between different apps, please make sure that the UIDs in the model JSON file (e.g. objectbox-default.json) are the same everywhere.

    How to run the examples

    Here you’ll find requirements and step-by-step guides for running the task-list example app in each of the three languages.

    C++ example app

    Requirements

    New to C++? Check out our beginner C++ ObjectBox installation tutorial.

    • WSL Ubuntu
    • CMake
    • Git
    • C++
    • Clang

      Step-by-step guide

      1.Start by creating a CMakelists.txt file:

      Now configure and build the project via CMake: Configure (Clang), CMake: Build.

      2. Sync-enabled objects: note the first line in tasklist.fbs.

      3. [if not running a server already] Start the ObjectBox Sync Server on Linux by running ./sync-server --model build/_deps/objectbox-src/examples/cpp-gen/objectbox-model.json --unsecured-no-authentication

      where sync-server is the path to your sync server executable. You can find more information about the server in the Sync Server docs.

      4. Sync Client: launch [objectbox-c-examples-cpp-gen-sync], and the Sync Client will start automatically. You can see how it was implemented in main.cpp.

      As this is just an example, we opted for no authentication to make things simple. This is not what you would use in production. We currently offer two authentication methods: shared secret and Google Sign-In. Here is the relevant Sync docs section on authentication options that explains how to use these.

      5. Let’s add a first task, called “task-cpp” (new task-cpp-1), to check if our C++ app syncs correctly. The output should look like this:

      Output of the C++ tasklist example app, showing a newly added task

      6. You can finally open the Admin UI to check if the task appears there. This is most easily done by opening http://127.0.0.1:9980/ in any web browser. For a more detailed description of what this can do, check out the Admin UI docs.

      Go example app

      Requirements

      • WSL Ubuntu
      • Go (see how to configure it for VS Code here)
      • Git

      Step-by-step guide

      1. First, clone the objectbox-go repository to your VS Code project. Make sure the current directory is objectbox-go.

      2. Sync-enabled objects. There are two versions of the task-list example: with and without sync. To run the one with sync, we need to enable our Task object for syncing. To do this, simply put the sync annotation on a new line in examples/tasks/internal/model/task.go:

      Then run the generator: go generate examples/tasks/internal/model/task.go to update the schema.

      3. [if not running a server already] Now start the ObjectBox Sync Server: ./sync-server --model=examples/tasks/internal/model/objectbox-model.json --unsecured-no-authentication,

      where sync-server is the path to your sync server file. You can find more information about the server in the Sync Server docs.

      4. Run go run examples/tasks/main.go. The Sync Client will start within the app; check main.go to see how this was implemented.

      As this is just an example, we opted for no authentication to make things simple. This is not what you would use in production. We currently offer two authentication methods: shared secret and Google Sign-In. Here is the relevant Sync docs section on authentication options that explains how to use these.

      5. Now we can add our first task (new task-go) – if it synced correctly, you should already see that from the output of the app. In particular, there will be a message from the change listener (“received 1 changes”):

      Output of the Go task-list example app after adding a first task

      6. Lastly, open the Admin UI to check if the task appears there. This is most easily done by opening http://127.0.0.1:9980/ in any web browser. For a more detailed description of what this can do, check out the Admin UI docs.

      Admin UI showing a task created with the Go example app

      Java (Android) example app

      Requirements

      • Java
      • Android Studio

      Step-by-step guide

        1. First of all, open Android Studio and clone the objectbox-examples repository via File → New → Project from Version Control. Use this URL: https://github.com/objectbox/objectbox-examples.git
        2. Sync-enabled objects: check out Task.java to see how this was done (note the @Sync annotation).
        3. [if not running a server already] Start the ObjectBox Sync Server

      ./sync-server --model android-app-sync/objectbox-models/default.json --unsecured-no-authentication,

      where sync-server is the path to your sync server file. You can find more information about the server in the Sync Server docs.

      1. Now you can run “android-app-sync” on a device of your choice. The Sync Client will start in the app. 

      As this is just an example, we opted for no authentication to make things simple. This is not what you would use in production. We currently offer two authentication methods: shared secret and Google Sign-In (only for Java, Kotlin, Dart, C & Go). Here is the relevant Sync docs section on authentication options that explains how to use these.

      5. Add a new task called “task-java”.

      6. Finally, open the Admin UI to check if the task appears there. This is most easily done by opening http://127.0.0.1:9980/ in any web browser. For a more detailed description of what this can do, check out the Admin UI docs.

      Next Steps

      How easy was that? cool Now that you’ve run your first ObjectBox Sync example, why not build something yourself? Use any combination of the supported languages to build your own cross platform app.

      We’re eager to see your use case examples! Don’t hesitate to share your results with us by posting on Social Media and tagging @objectbox_io, or simply sending us an email on contact[at]objectbox.io. 

       

      If you want to learn more about how ObjectBox can be used in IoT, here is an overview of different use cases

      What is Data Synchronization + How to Keep Data in Sync

      What is Data Synchronization + How to Keep Data in Sync

      What is Data Sync / Data Synchronization in app development?

      Data Synchronization (Sync) is the process of establishing consistency and consolidation of data between different devices. It is fundamental to most IT solutions, especially in IoT and Mobile. Data Sync entails the continuous harmonization of data over time and typically is a complex, non-trivial process. Even corporates struggle with its implementation and had to roll back Data Sync solutions due to technical challenges. 

      The question Data Sync answers is

      phone-data-sync-with-machine-payment-automatic-data

      How do you keep data sets from two (or more) data stores / databases – separated by space and time – mirrored with one another as closely as possible, in the most efficient way?

      Data Sync challenges include asynchrony, conflicts, slow bandwidth, flaky networks, third-party applications, and file systems that have different semantics.

      Data Sync versus Data Replication in Databases

      sync-data-better-than-replication

      Data replication is the process of storing the same data in several locations to prevent data loss and improve data availability and accessibility. Typically, data replication means that all data is fully mirrored / backed up / replicated on another instance (device/server). This way, all data is stored at least twice. Replication typically works in one direction only (unidirectional); there is no additional logic to it and no possibility of conflicts.

      In contrast, Data Sync typically relates to a subset of the data (selection) and works in two directions (bi-directional). This adds a layer of complexity, because now conflicts can arise. Of course, if you select all data for synchronisation into one direction, it will yield the same result as replication. However, replication cannot replace synchronization.

      Why do you need to keep data in sync?

      Think about it – if clocks were not in sync, everyone would live on a different time. While I can see an upside to this, it would result in many inefficiencies as you could not rely on schedules. When business data is not in sync (up-to-date everywhere), it harms the efficiency of the organization due to:

      • Isolated data silos
      • Conflicting data / information states
      • Duplicate data / double effort
      • Outdated information states / incorrect data

      In the end, the members of such an organization would not be able to communicate and collaborate efficiently with each other. They would instead be spending a lot of time on unnecessary work and “conflict resolution”. On top, management would miss an accurate overview and data-driven insights to prioritize and steer the company. The underlying mechanism that keeps data up-to-date across devices is a technical process called data synchronization (Sync). And while we expect these processes to “just work”, someone needs to implement and maintain them, which is a non-trivial task.

      Growing data masses and shifts in data privacy requirements call for sensible usage of network bandwidth and the cloud. Edge computing with selective data synchronization is an effective way to manage which data is sent to the cloud, and which data stays on the device. Keeping data on the edge and synchronizing selective data sets effectively, reduces the data volume that is transferred via the network and stored in the cloud. Accordingly, this means lower mobile networking and cloud costs. On top, it also enables higher data security and data privacy, because it makes it easy to store personal and private data with the user. When data stays with the user, data ownership is clear too.

      Unidirectional Data Replication

      replication-data-sync-database

      Bidirectional Data Synchronization

      how-to-sync-data-what-is-data-sync

      Out-of-the-box Sync magic: Syncing is hard

      Almost every Mobile or IoT application needs to sync data, so every developer is aware of the basic concept and challenges. This is why many experienced developers appreciate out-of-the-box solutions. While JSON / REST offers a great concept to transfer data, there is more to Data Sync than what it looks like at a glance. Of course, the complexity of Sync varies widely depending on the use case. For example, the amount of data, data changes, synchronous / asynchronous sync, and number of devices (connections), and what kind of client-server or peer-to-peer setup is needed, all affect the complexity.

      iceburg-building-data-synchronization

      What looks easy in practice hides a complex bit of coding and opens a can of worms for testing. For an application to work seamlessly across devices – independent of the network, which can be offline, flaky, or only occasionally connected – an app developer must anticipate and handle a host of local and network failures to ensure data consistency. Moreover, for devices with restricted memory, battery and/or CPU resources (i.e. Mobile and IoT devices), resource sensitivity is also essential. Data storage and synchronization solutions must be both effective / efficient, and sustainable.

      How to Keep Data in Sync Without the Headache?

      Thankfully, there are out-of-the-box data synchronization solutions available on the market, which solve data syncing for developers. They fall broadly into two categories: cloud-dependent data synchronization, and independent, “edge” data synchronization. Cloud-based solutions, like Firebase, require a connection to the internet to function. Data is sent to and requested from the cloud constantly. Edge solutions, like ObjectBox, also offer “Offline Sync”: Data is stored in an efficient on-device database, synchronization on and between edge devices can be done continually without an Internet connection, and Dat Sync with a cloud or a backend that is not located on premise occurs once the device(s) goes online. Below, we summarize the most popular market offerings for data synchronization (offline and cloud based):

      mongo-realm-logo

      Couchbase

      Couchbase is a Cloud DB, Edge DB and Sync offering that requires the use of Couchbase servers.

      firebase-logo

      Firebase

      Firebase is a Backend as a Service (BaaS) offering from Google (acquired). Google offers it as a cloud hosted solution for mobile developers.

      mongo-realm-logo

      Mongo Realm

      Realm was acquired by MongoDB in 2019; the Mongo Realm Sync solution is now in Alpha and available hosted with MongoDB.

      mongo-realm-logo

      ObjectBox

      ObjectBox is a DB for any device, from restricted edge devices to servers, and offers an out-of-the-box Sync solution. ObjectBox enables self-hosting on-premise / in the cloud, as well as Offline Sync.

      pasre-logo-comparison

      Parse

      Parse is a BaaS offering that Facebook acquired and shut down. Facebook open sourced the code. The GitHub repository is not officially maintained. You can host Parse yourself or use a Parse hosting service.

      Data Sync, Edge Computing, and the Future of Data

      There is a megashift happening in computing from centralized cloud computing to Edge Computing. Edge computing is a decentralized topology entailing storing and using data as close to the source of the data as possible, i.e. directly on edge devices. Accordingly, the market is growing rapidly with projections estimating continuing growth with a 34% CAGR for the next five years. The move from the cloud to the edge is strongly driven by new use cases and growing data masses Edge data persistence and Data Sync (managing decentralized data flows), especially “Offline Sync”, are the key technologies needed for Edge Computing. Using edge data persistence, data can be stored and processed on the edge. This means application always work, independent from a network connection, offline. Faster response times can be guaranteedWith Offline Sync, data can be synchronized between several edge devices in any location independant from an Internet connection. Once a connection becomes available, selected data can be synchronized with  a central server. By exchanging less data with the cloud or a central instance, data synchronization reduces the burden on the network. This brings down mobile network and cloud costs, and reduces the amount of energy used: a win-win-win solution. It also enables data privacy by design.

      Flutter databases –  hive, ObjectBox, sqflite (+ Drift, floor)

      Flutter databases – hive, ObjectBox, sqflite (+ Drift, floor)

      Flutter, the renowned cross-platform mobile framework, has been gaining immense popularity among developers worldwide. As the Flutter community expands, the demand for efficient Flutter databases is also increasing. Developers now have access to a range of Flutter database options that cater to various needs and preferences.

      In this article, we’ll focus specifically on local storage solutions, as these are essential for enabling offline functionality, improving performance, ensuring data persistence, enhancing data privacy and security, and supporting edge computing capabilities. Furthermore, local data storage is needed to promote sustainability. Let’s dive into the current local database landscape for Flutter and compare the most popular options.

      Flutter databases / Flutter Dart data persistence

      While the database market is huge and dynamic,  there are only few options to choose from if you are a Flutter / Dart app developer. Before we dive into the Flutter database options, advantages and disadvantages, we’re taking a very quick look at databases to make sure, we share a common ground. 

      What is a database?

      A database is a piece of software that allows the storage and systematic use of digital information, in other words: data persistence. As opposed to mere caching, data is reliably stored and available to work with unless actively deleted. A database typically allows developers to store, access, search, update, query, and otherwise manipulate data in the database via a developer language or API. These types of operations are done within an application, in the background, typically hidden from end users. Many applications need a database as part of their technology stack. The most typical database operations are CRUD: Create, Read, Update, Delete.

      What are the major types of databases?

      There are many types of databases. For our purpose, the most important differentiations are non-relational (NoSQL) versus relational databases (SQL), cloud databases versus edge databases, and maybe embedded versus in-memory. However, databases can be further distinguished by additional criteria e.g. the data types they support, or the way they scale – and definitions can vary.

      What is an ORM?

      An Object relational Mapper (ORM) is not a database. We’re bringing this up mainly, because we see it confused often. It is a layer that sits on top of a database and makes it easier to use. This is typically especially relevant when the database is a relational database (SQL) and the programming language used is object-oriented. As noted above, Dart is an object-oriuented programming language.

      The Flutter local data persistence landscape

      There are several Flutter databases that provide offline support, offering the ability to store and access data locally even without an internet connection. Here are some of the notable options:

      • Hive is a lightweight key-value database written in Dart for Flutter applications, inspired by Bitcask.
      • ObjectBox DB is a highly performant lightweight NoSQL database with an integrated Data Sync. It stores objects.
      • sqflite is a wrapper around SQLite, which is a relational database without direct support for Dart objects. 
      • Drift is a reactive persistence library for Flutter and Dart, built ontop of SQLite. 
      • Floor is another ORM on top of SQLite.

       

      What is the best offline Flutter Dart database?

      This of course depends… Make up your own mind with the following comparison matrix as a starting point. Note: With very few options to choose from, the following overview is sometimes a bit like comparing apples 🍎 and pears 🍐.

      Name Description Primary Model Language License Data Sync
      Hive Lightweight key-value database NoSQL Dart Apache 2.0
      ObjectBox Lightweight NoSQL database with integrated Data Sync NoSQL Dart Bindings are Apache 2.0
      Drift ORM on top of SQLite relational SQL SQLite is public domain, Drift is MIT
      Floor ORM on top of SQLite relational SQL SQLite is public domain, Floor is Apache 2.0
      sqflite SQLite plugin for Flutter relational SQL SQLite is public domain, sqflite lib is MIT
      <p>Diese Seite verwendet Frames. Frames werden von Ihrem Browser aber nicht unterstützt.</p>

      Flutter Database performance benchmarks

      As with any benchmark, you need to take a look at the details. We take benchmarking very serious and strive to get accurate results. Therefore, we also always open source the benchmarking code and encourage you to check it out. If you note anything that does not even out in your oppinion, do let us know. We have a long history of updating and improving our benchmarks continually and are happy to take any recommendations.

      Performance Benchmark Test Setup

      We used an Android 10 device with a Kirin 980 CPU to run the benchmarks as a Flutter app. The app executed all operations (ops) in batches of 10.000 objects. Each batch formed a single transaction. We ran each test 50 times. The results you see in the diagram are averages across all runs. We set it up that way to ensure that neither the Virtual Machine warmup during the first run nor the garbage collections affect the overall result significantly. 

      Flutter Databases CRUD Performance Results

      Summary of the Flutter Dart DB Benchmarks

      Hive and ObjectBox clearly outperform sqflite across all CRUD operations. The results show ObjectBox performing with up to 70 times the speedup for create and update operations. With regards to comparing Hive and ObjectBox, the results vary more. Hive can be faster at reading objects than ObjectBox. However, strictly speaking it’s not a fair comparison, because in Hive, the high read numbers result from Dart objects already cached in memory. If the objects are fetched using the async API from disk, the numbers drop by factor 1000.

      Drift and Floor were not part of the benchmarking as they are ORMs. However, it is very likely they will perform similarly to sqflite, reflecting primarily the performance of SQLite.

      Flutter Data persistence – Conclusion

      Recently, the Flutter database landscape has experienced significant growth and diversification. With Flutter’s increasing popularity, developers now have a number of database options available. In this article, we focused on the best local databases, comparing their features in a comprehensive matrix, and showcasing performance benchmarks. In the end, the best choice depends on the specific needs of each project. The Flutter database landscape in 2023 is a thriving ecosystem, continuously evolving to meet the changing needs of Flutter app development. One upcoming change that we can see is the rise of vector databases for AI. So, we encourage you to keep an eye on the lively market of Flutter databases not to miss any important updates.

      If you want to get started learning how to use a database, we suggest you check out this video tutorial series that teaches you how to build a Flutter app with ObjectBox from scratch.

       

      Firebase alternatives for Data Storage and Data Sync

      Firebase alternatives for Data Storage and Data Sync

      Data Sync is a typical recurring and typically non-trivial developer challenge. Synchronizing data in offline/online settings, like for example across eventually connected devices, is simply hard. While JSON / REST is great, building Data Sync yourself is time-consuming, risky, and typically considered no fun. Therefore, today, we take a look at the out-of-the-box Data Sync market. If you are rather interested learning about Data Sync in general, check out this article about why data sync technology is more necessary than ever.

      Introduction

      One of the most well-known Data Sync solutions is Firebase. However, Firebase is purely cloud based and offers no support for local data storage ( as in “data persistence above caching”) and therefore offline usage. With a huge shift happening in computing from the cloud to the edge, offline-first approaches and Edge Computing are getting more and more important. Therefore, we’ve recently taken a comprehensive look at mobile database and edge database offerings on the market. But what options do Mobile and IoT developers working on the edge have for out-of-the-box Data Sync solutions? Very few. While there are more and more cloud-based Firebase alternatives springing up nearly daily (e.g. appwrite and supabase) forcing the user into a centralized cloud setup, there is almost nothing that supports offline Data Sync and / or persistent local data storage. As our focus is on offline / edge Data Sync and local storage, in the following we add all edge / offline Data Sync solutions we know of, but spare you the wealth of cloud options only adding the established ones.

      Firebase

      Firebase is a cloud backend service ((Mobile) Backend as a Service ((M)BaaS)) that enables developers to build mobile or web applications without needing to take care of the backend. This includes the data synchronization, scalability, network, infrastructure challenges etc. Indeed, Firebase, today, offers many different services (e.g. analytics, crashlytics) and goes well beyond Data Sync. We are looking at Firebase from the Data Sync perspective only. Firebase was one of the first Data Sync solutions available on the market together with Parse and Couchbase, which all started in 2011 (Couchbase through a merger of CouchOne and Membase). In 2014, Firebase was acquired by Google. Incidentally, the same year Parse was acquired by Facebook to be subsequently shut down, and Couchbase raised significant funding. All three are still in use today. 

      Firebase Pros and Cons

      In the following, we will first look at the advantages and disadvantages of Firebase. Then, we will compare Firebase with Firebase alternatives like Couchbase, Parse and ObjectBox in a comprehensive matrix.  

      Firebase Advantages ++

      Firebase Disadvantages —

      Cloud based Purely cloud based
      Google: large team that supports and maintains it; very low risk of the company failing; however, Google has a reputation of discontinuing products / services, so there is no guarantee Google: vendor lock-in (no migration tools prevents you from making your app portable), you cannot access your data as it is hosted on the Firebase server
      Backend as a service (ease of use) Less flexibility: You cannot optimize the backend to match your app’s needs

      The Firebase Realtime Database has its own advantages:

      • hosted, powered by Google
      • for pure online use cases rather fast
      • great if you do not have a strong DB background

      The Firebase Realtime Database has its own drawbacks:

      • the whole DB is a huge JSON file
      • limited querying capabilities
      • no way to efficiently filter data
      • Easily disorganized, hard to navigate and search
      Pay as you go, price scales with usage Cost insecurities, hard to impossible to predict
      Less iOS support (stronger focus on Android) Less iOS support (stronger focus on Android)
      Doesn’t work in countries that don’t allow Google
      User privacy concerns***

       

      *** “Firebase has been claimed to be used by Google to track users without their knowledge. On July 14, 2020, a lawsuit was filed accusing Google of (…) logging what the users are looking at in many types of apps, despite the user following Google’s own instructions to turn off the web and app activity collected by the company.” (https://en.wikipedia.org/wiki/Firebase)anced settings.

      Firebase Advantages ++

      Cloud based
      – Google: large team that supports and maintains it; very low risk of the company failing; however, Google has a reputation of discontinuing products / services, so there is no guarantee
      – Backend as a service (ease of use)
      – The Firebase Real-time Database has its own advantages:
      – Pay as you go, price scales with usage
      – Less iOS support (stronger focus on Android)

      Firebase Disadvantages —

      – Purely cloud based
      Google: vendor lock-in (no migration tools prevents you from making your app portable), you cannot access your data as it is hosted on the Firebase server
      – Less flexibility: You cannot optimize the backend to match your app’s needs
      The Firebase Real-time Database has its own drawbacks
      the whole DB is a huge JSON file
      limited querying capabilities
      no way to efficiently filter data
      Easily disorganized, hard to navigate and search
      – Cost insecurities, hard – impossible to predict
      – Less iOS support (stronger focus on Android)
      – Doesn’t work in the countries that don’t allow Google
      User privacy concerns: “Firebase has been claimed to be used by Google to track users without their knowledge. On July 14, 2020, a lawsuit was filed accusing Google of (…) logging what the users are looking at in many types of apps, despite the user following Google’s own instructions to turn off the web and app activity collected by the company.” (https://en.wikipedia.org/wiki/Firebase)

      Firebase alternatives: A look at out-of-the-box data sync solutions

      The majority of offerings for developers that handle Data Sync as defined here, are cloud-based and fall into the category of BaaS (can also be MBaaS (Mobile Backend as a Service) or PaaS (platform as a Service) or DBaaS (Database as a Service). This means that data synchronisation is only a specific part of the whole offering. 

      Data Sync Solution comparison matrix – Firebase and its alternatives

      Solution name Company Category Data Sync IoT / Mobile Database Type of DB Cloud OS / Platforms Languages License
      Cloudant Sync
      IBM (Cloudant was acquired in 2014) DBaaS
      (Cloud DB and Cloud Sync)
      Two-way
      cloud data replication (called “sync”)
      IoT
      & Mobile
      Cloud
      database based on Couch DB
      NoSQL;
      distributed JSON document database
      Cloud-based
      replication to and from on-device data (CouchDB <> cloud service)
      hosted
      service
      C#,
      Java, JavaScript, Objective-C, PHP, Ruby
      Proprietary
      (CouchDB is Apache 2.0 and they integrate with several open source libraries)
      Couchbase server &  Sync Gateway
      Couchbase (a merger of Couch One and Membase) Cloud
      DB and Cloud Sync
      Sync
      needs a Couchbase Server
      IoT
      & Mobile
      Edge:
      Couchbase Lite; Server: Couchbase
      NoSQL;
      document database
      Always
      needs Couchbase Server (originally Membase)
      mainly
      used as hosted service;
      iOS, Android, .NET (Desktop/Server), .NET UWP, Xamarin
      Swift,
      Objective-C, Java (Android), Java (Non-Android), Kotlin, C#, JavaScript, C
      Apache
      2.0, delayed open source
      Firebase**
      Google (Firebase was acquired by Google in 2014) BaaS
      (Cloud)
      Cloud
      Sync via Google servers
      Mobile Cloud:
      Firebase Realtime Database; Edge: Caching only (Firestore)
      Document
      store
      hosted
      only
      APIS
      for iOS & Android
      JavaScript API
      RESTful HTTP API
      Java
      JavaScript
      Objective-C
      proprietory
      Mongo Realm Sync
      MongoDB
      (Realm was acquired in 2019)
      Cloud
      DB and Cloud Sync
      Sync
      (in Alpha); only via Mongo Cloud
      IoT
      & Mobile
      Cloud:
      MongoDB, Edge: Mongo Realm
      MongoDB:
      NoSQL document store; RealmDB: Embedded NoSQL DB
      hosted
      service
      MongoDB:
      Linux, OS X, Solaris, Windows
      Mongo Realm DB:
      Android, iOS
      20+
      languages, e.g. Java, C, C#, C++
      Mongo
      DB changed its license from open source (GNU) to MongoDB Inc.’s Server Side
      Public License (SSPL) in 2018.
      ObjectBox
      Sync
      ObjectBox DB
      and Sync

      Offline
      Sync, on-premise Sync, Cloud Sync

      p2p Sync is planned

      IoT
      & Mobile
      ObjectBox Object-oriented
      embedded NoSQL DB
      Self-hosted
      / on-premise; hosted service upon request only
      iOS,
      Android, Linux, Windows, MacOS, any POSIX-system
      C,
      C++
      Java
      Kotlin
      Swift
      Go
      Flutter / Dart
      Python
      DB:
      Open source bindings, Apache 2.0, proprietary core
      Parse
      Originally
      Parse, acquired by Facebook, closed down and open sourced, unmaintained
      MBaaS
      (Cloud)
      Cloud
      Sync, self-hosted or via a provider that offers Parse hosting
      Mobile Both,
      PostgreSQL* and MongoDB, can be used as a database for Parse
      MongoDB:
      NoSQL document store; PostgreSQL:
      Only
      Cloud, only self-hosted or via a provider that offers Parse hosting
      Server: REST
      API lets you interact with Parse Server from anything that can send an HTTP
      request
      open
      source, BSD
      Syncstudio
      HandApps Cloud-based
      sync between SQLite and MS SQL Server based in the MS Sync Framework
      Sync Mobile Edge:
      SQLite or MSSQL (including LocalDB or Express); Server:
      Microsoft SQL
      relational
      / SQL
      SQL
      Server; Sync / replication works via cloud only
      Android
      Java, Basic4Android, Windows Forms, UWP, Windows Mobile, Xamarin
      proprietory,
      4 licenses available: Community/Free, Subscription, Perpetual and Royalty
      Free
      Zumero
      Zumero
      LLC
      Cloud-based
      replication of SQL data for Mobile
      Sync Mobile Edge:
      SQLite; Server: Microsoft SQL
      relational
      / SQL
      SQL
      Server; Sync / replication works via cloud only
      Mobile
      only (iOS, Android, Xamarin, PhoneGap)

      proprietory,
      annual license scaling with the number of devices

       

       

       

       

      Notes: Microsoft Sync Framework (renamed Sync Framework Toolkit at some point) is a legacy open source product which MS no longer supports

      * PostgreSQL vs Postgres
      ** There are many Cloud Sync alternatives to Firebase, we added the more prominent options and any service that also serves Edge Computing

      Data Sync is no standardized term and though it seems to be in use by many big companies and most dvelopers will have a notion of what it is, the devil is in the details. So, we might have missed an important solution or taken an angle someone else would not agree with. Please feel free to let us know what to improve.

      👉 Want to save this info for later? Watch the Firebase alternatives matrix on GitHub to find it easily wherever you need it.

      ObjectBox DB and Sync – designed to keep data up to date across time and space

      ObjectBox is a high performance NoSQL fully ACID-compliant edge database built from scratch for efficient data on and across restricted and occasionally connected devices, taking care of keeping data in sync reliably. ObjectBox developer tools are easy to use, quick to implement, and optimized for high-performance and frugal resource-use on edge devices running mobile, desktop, server, and IoT applications. ObjectBox helps developers to focus on what they love and build great applications without needing to take care of the boilerplate code for resilient connectivity, synchronizing data, and tedious DB optimizations. This cuts down initial implementation efforts, ongoing maintenance efforts, undesired problems, and data loss – therefore reducing costs and time to market tremendously. We are dedicated to bring joy and delight to Mobile and IoT application developers.

      How Building Green IoT Solutions on the Edge Can Help Save Energy and CO2

      How Building Green IoT Solutions on the Edge Can Help Save Energy and CO2

      The internet of things (IoT) has a huge potential to reduce carbon emissions, as it enables new ways of operating, living, and working [1] that are more efficient and sustainable. However, IoT’s huge and growing electricity demands are a challenge. This demand is due primarily to the transmission and storage of data in cloud data centers. [2] While data center efficiency and the use of green energy will reduce the CO2 emissions needed for this practice, it is not addressing the problem directly. [3

      iot-data-cloud-energy-waste

      With ObjectBox, we address this unseen and fast-growing CO2 source at the root: ObjectBox empowers edge computing, reducing the volume of data transmitted to central data storage, while at the same time, heightening data transmission and storage efficiency. [4] We’ve talked before about how edge computing is necessary for a sustainable future, below we dive into the numbers a bit deeper. TLRD: ObjectBox enables companies to cut the power consumption of their IoT applications, and thus their emissions, by 50 – 90%. For 2025, the potential impact of ObjectBox is a carbon emission reduction of 594 million metric tons (see calculations below).

      How ObjectBox’ Technology Reduces Overall Data Transmission

       ObjectBox reduces data transmission in two ways: 1. ObjectBox reduces the need for data transmission, 2. ObjectBox makes data transmission more efficient. ObjectBox’ database solution allows companies to build products that store and process data on edge devices and work with that data offline (as well as online). This

      Green IoT Solution

      not only improves performance and customer experience, it also reduces the overall volume of data that is being sent to the cloud, and thus the energy needed to transfer the data as well as store it in the cloud. ObjectBox’ Synchronization solution makes it easy for companies to transmit only the data that needs to be transmitted through 1) selective two-way syncing and 2) differential delta syncing. Synchronizing select data reduces the energy required for unnecessarily transmitting all data to the cloud.

      We have demonstrated in exemplary case studies that ObjectBox can reduce total data transmissions by 70-90%, depending on the case. There will, however, typically be value in transmitting some parts of data to a central data center (cloud); ObjectBox Sync combines efficient compression based on standard and proprietary edge compression methods to keep this data small. ObjectBox also has very little overhead. Comparing the transmission of the same data sets, ObjectBox saves 40-60% on transmission data volume through the delta syncing and compression, and thus saves equivalent CO2 emissions for data transmissions. Additional studies support these results, and have shown that moving from a centralized to a distributed data structure, saves between 32 and 93% of transmission data. [5

      sync-sustainable-data-save-energy

      Calculations: How Does ObjectBox Save CO2?

      Physically using a device consumes little energy directly; it is the wireless cloud infrastructure in the backend (data center storage and data transmission) that is responsible for the high carbon footprint of mobile phones [6] and IoT devices. Estimates say that IoT devices will produce around 2,8 ZB of data in 2020 (or 2,823,000,000,000  GB), globally. [7] Only a small portion of that data actually gets stored and used; we chose to use a conservative estimate of 5% [8] (141,150,000,000 GB) and of that portion, 90% is transferred to the cloud [9] (127,035,000,000 GB). Transferring 1 GB of data to the cloud and storing it there costs between 3 and 7 kWh. [10] Assuming an average of 5 kWh this means a 127,035,000,000 GB multiplied by 5kWh, resulting in a total energy expenditure of 635,175,000,000 kWh. Depending on the energy generation used, CO2 emissions vary. We are using a global average of 0,475 kgCO2 / 1 kwH. [11] In total this means that there will be 301,708,125,000 KG of CO2, or roughly 301 million metric tons of CO2 produced to transfer data to the cloud and store it there in 2020. 

      Projections for 2025 have data volumes as high as 79.4 ZB. [12] Following the same calculations as above, IoT devices would be responsible for 8 billion metric tons of CO2 in 2025.* We estimate that using ObjectBox can cut CO2 caused by data transmission and data centers by 50-90%, by keeping the majority of data on the device, and transmitting data efficiently. It will take time for ObjectBox to enter the market, so assuming a 10% market saturation by 2025 and an average energy reduction of 70%, using ObjectBox could cut projected CO2 emissions by 594 million metric tons in 2025.

      ObjectBox is on a mission to reduce digital waste which unnecessarily burdens bandwidth infrastructure and fills cloud servers, forcing the expansion of cloud farms and in turn, contributing to the pollution of the environment. As our digital world grows, we all need to give some thought to how we should structure our digital environments to optimize and support useful, beneficial solutions, while also keeping them efficient and sustainable. 

      *Of course, in that time, the technologies will all be more efficient and thus use less electricity while at the same time CO2 emissions / kWh will have dropped too. Thus, we are aware that this projection is an oversimplification of a highly complex and constantly changing system.

      [1] https://www.theclimategroup.org/sites/default/files/archive/files/Smart2020Report.pdf
      [2] https://www.iea.org/reports/tracking-buildings/data-centres-and-data-transmission-networks
      [3]“Data centres… have eaten into any progress we made to achieving Ireland’s 40% carbon emissions reduction target.” from https://www.climatechangenews.com/2017/12/11/tsunami-data-consume-one-fifth-global-electricity-2025/
      [4] https://medium.com/stanford-magazine/carbon-and-the-cloud-d6f481b79dfe
      [5] https://www.researchgate.net/publication/323867714_The_carbon_footprint_of_distributed_cloud_storage
      [6] https://www.resilience.org/stories/2020-01-07/the-invisible-and-growing-ecological-footprint-of-digital-technology/
      [7] https://www.idc.com/getdoc.jsp?containerId=prUS45213219, https://priceonomics.com/the-iot-data-explosion-how-big-is-the-iot-data/, https://www.gartner.com/en/newsroom/press-releases/2018-11-07-gartner-identifies-top-10-strategic-iot-technologies-and-trends, https://www.iotjournaal.nl/wp-content/uploads/2017/02/white-paper-c11-738085.pdf, ObjectBox research
      [8] Forrester (https://internetofthingsagenda.techtarget.com/blog/IoT-Agenda/Preventing-IoT-data-waste-with-the-intelligent-edge), Harvard BR (https://hbr.org/2017/05/whats-your-data-strategy), IBM (http://www.redbooks.ibm.com/redbooks/pdfs/sg248435.pdf), McKinsey (https://www.mckinsey.com/business-functions/mckinsey-digital/our-insights/the-internet-of-things-the-value-of-digitizing-the-physical-world)
      [9] https://www.gartner.com/smarterwithgartner/what-edge-computing-means-for-infrastructure-and-operations-leaders/
      [10] According to the American Council for an Energy-Efficient Economy: 5,12 kWh of electricity / GB of transferred data. According to a Carnegie Mellon University study: 7 kWh / GB. The American Council for an Energy-Efficient Economy concluded: 3.1 kWh / GB.
      [11] https://www.iea.org/reports/global-energy-co2-status-report-2019/emissions
      [12] https://www.idc.com/getdoc.jsp?containerId=prUS45213219