Flutter databases –  sqflite, hive, ObjectBox, and Moor

Flutter databases – sqflite, hive, ObjectBox, and Moor

Flutter is becoming a serious developer platform and with it grows a need for Flutter databases. A quick note on Flutter and Dart as it is pretty young and you might either be looking for a Flutter database, a Dart database or truly a Flutter Dart database. Flutter is an open-source UI software development kit created by Google. Dart is the programming language in which developers code Flutter apps. Dart is an object-oriented programming language. 

Flutter databases / Flutter Dart data persistence

While the database market is huge and dynamic,  there are only few options to choose from if you are a Flutter / Dart app developer. Before we dive into the Flutter database options, advantages and disadvantages, we’re taking a very quick look at databases to make sure, we share a common ground. Accordingly, we’ll not get theoretical and extensive, but focus on what we mean here. 

What is a database?

A database is a piece of software that allows the storage and systematic use of digital information, in other words: data persistence. A database typically allows developers to store, access, search, update, query, and otherwise manipulate data in the database via a developer language or API. These types of operations are done within an application, in the background, typically hidden from end users. Many applications need a database as part of their technology stack. The most typical database operations are CRUD: Create, Read, Update, Delete.

What are the major types of databases?

There are many types of databases. For our purpose, the most important differentiations are non-relational (NoSQL) versus relational databases (SQL), cloud databases versus edge databases, and maybe embedded versus in-memory. However, databases can be further distinguished by additional criteria e.g. the data types they support, or the way they scale – and definitions can vary.

What is an ORM?

An Object relational Mapper (ORM) is not a database. We’re bringing this up mainly, because we see it confused often. It is a layer that sits on top of a database and makes it easier to use. This is typically especially relevant when the database is a relational database (SQL) and the programming language used is object-oriented. As noted above, Dart is an object-oriuented programming language.

The Flutter Dart data persistence landscape

At this point in time, the database landscape for Flutter Dart is still very limited. So, let us quickly introduce the current market players. Note: We are adding in Moor, because with that few player it is just one more option available and therefore, at this moment in time, should be part of the Flutter Dart data persistence landscapes, in our minds.

  • Firebase Realtime DB is a cloud-hosted database. It stores data as JSON and synchronizes it to connected clients.
  • Hive is a lightweight key-value database written in Dart for Flutter applications, inspired by Bitcask.
  • ObjectBox DB is a highly performant lightweight NoSQL database. It stores objects.
  • sqflite is a wrapper around SQLite, which is a relational database without direct support for Dart objects. 
  • Moor is a reactive persistence library for Flutter and Dart, built ontop of sqlite. 

 

What is the best Flutter Dart database?

This of course depends… Make up your own mind with the following comparison matrix as a starting point. Note: With very few options to choose from, the following overview is sometimes a bit comparing apples🍎 and pears🍐.

 

Data persistence Description Primary Model Location of data Language License Fun Fact
Firebase Realtime Database Mobile Backend as a Service (MBaaS) NoSQL Google Cloud Dart Proprietary acquired by Google in 2014
hive Light key-value DB for Flutter NoSQL local Dart Apache 2.0 Munich brew
ObjectBox High-performance Flutter DB NoSQL local, self-hosted server / cloud Dart Bindings are Apache 2.0 Munich-brew out-of-the-box data sync solution
sqflite SQLite plugin for Flutter relational local SQL SQLite is public domain, sqflite lib is MIT good old SQLite
Moor ORM for SQLite used on top of a relational DB local Dart SQLite is public domain, Moor lib is MIT Room spelled backwards

 

<body> <p>Diese Seite verwendet Frames. Frames werden von Ihrem Browser aber nicht unterstützt.</p> </body>

Flutter Database performance benchmarks

As with any benchmark, you need to take a look at the details. We take benchmarking very serious and strive to get accurate results. Therefore, we also always open source the benchmarking code and encourage you to check it out. If you note anything that does not even out in your oppinion, do let us know. We have a long history of updating and improving our benchmarks continually and are happy to take any recommendations.

Performance Benchmark Test Setup

We used an Android 10 device with a Kirin 980 CPU to run the benchmarks as a Flutter app. The app executed all operations (ops) in batches of 10.000 objects. Each batch formed a single transaction. We ran each test 50 times. The results you see in the diagram are averages across all runs. We set it up that way to ensure that neither the Virtual Machine warmup during the first run nor the garbage collections affect the overall result significantly. 

Flutter Databases CRUD Performance Results

Flutter Database Performance Benchmarks

Summary of the Flutter Dart DB Benchmarks

Hive and ObjectBox clearly outperform sqflite across all CRUD operations. The results show ObjectBox performing with up to 70 times the speedup for create and update operations. With regards to comparing Hive and ObjectBox, the results vary more. Hive can be faster at reading objects than ObjectBox. However, strictly speaking it’s not a fair comparison, because in Hive, the high read numbers result from Dart objects already cached in memory. If the objects are fetched using the async API from disk, the numbers drop by factor 1000.

As a cloud-based online database, Firebase is not really comparable. Local data persistence, an edge database, will typically always beat a cloud-based solutions with regards to response times. But of course cloud-based solutions have their own advantages and there may be reasons why you would choose to use Firebase over an edge database. It still may be a great option for you, depending on the use case.

Moor was not part of the benchmarking as it is an ORM. However, it is very likely it will perform similarly as sqflite, reflecting primarily the performance of SQLite.

Flutter and Data persistence landscape Conclusion

Flutter is becoming a serious developer platform and developers need a data persistence solution. There are currently only few databases supporting the Flutter community and 2021 will be an interesting year to watch where this is going. If you are interested to learn more about the database space, DB-engines and the database of databases are great starting points. Otherwise, go, check out ObjectBox Dart for Flutter Dart and share your thoughts with us – it’s not too late yet; we are shortly before releasing 1.0 and your feedback counts 🙂

Firebase, Firebase alternatives – a look at data sync solutions

Firebase, Firebase alternatives – a look at data sync solutions

Data Sync is one of the typical challenges a developer faces. Synchronizing data is not easy and while JSON / REST are great, building data sync yourself is challenging. Therefore, today, we take a look at the out-of-the-box data sync market. If you are rather interested learning about data sync in general, check out this article about why data sync technology is more necessary than ever.

As Firebase is one of the most well-known data sync solutions, we look at the market from a Firebase alternatives perspective. However, Firebase is purely cloud based and offers no support for local data storage and offline usage. With a huge shift happening in computing from the cloud to the edge, offline-first approaches and Edge Computing are getting more and more important. Reasons to move to the edge, include: Data Speed, Data Privacy/Security, Broadband Limitations, Network Costs, Cloud Costs, and User Experience. Therefore, we’ve recently taken a comprehensive look at mobile database and edge database offerings on the market, but what options do Mobile and IoT developers working on the edge have for out-of-the-box data synchronization solutions? Very few. As there is no lack of cloud-based offerings, we focus on the most prominent cloud services only, and add all edge / offline sync solutions, we are aware of, to our comprehensive comparison of data sync solutions. 

Firebase

Firebase is a cloud backend service ((Mobile) Backend as a Service ((M)BaaS)) that enables developers to build mobile or web applications without needing to take care of the backend, which includes the data synchronization, scalability, network, infrastructure challenges etc. Firebase, today, offers many different services (e.g. analytics, crashlytics) and goes well beyond sync; we are looking at Firebase from the sync perspective only. Firebase was one of the first data sync solutions available on the market together with Parse and Couchbase, which all started in 2011 (Couchbase through a merger of CouchOne and Membase). In 2014, Firebase was acquired by Google, Parse was acquired by Facebook to be subsequently shut down, and Couchbase went on to raise funding. All three are still in use today. 

Firebase Pros and Cons

In the following, we will look at the advantages and disadvantages of Firebase, and then compare Firebase with Firebase alternatives, like Couchbase, Parse and ObjectBox in a comprehensive matrix to give you a good overview of data sync options. 

Firebase Advantages ++ Firebase Disadvantages —
Cloud based Purely cloud based
Google: large team that supports and maintains it; very low risk of the company failing; however, Google has a reputation of discontinuing products / services, so there is no guarantee Google: vendor lock-in (no migration tools prevents you from making your app portable), you cannot access your data as it is hosted on the Firebase server
Backend as a service (ease of use) Less flexibility: You cannot optimize the backend to match your app’s needs

The Firebase Realtime Database has its own advantages:

  • hosted, powered by Google
  • for pure online use cases rather fast
  • great if you do not have a strong DB background

The Firebase Realtime Database has its own drawbacks:

  • the whole DB is a huge JSON file
  • limited querying capabilities
  • no way to efficiently filter data
  • Easily disorganized, hard to navigate and search
Pay as you go, price scales with usage Cost insecurities, hard to impossible to predict
Less iOS support (stronger focus on Android) Less iOS support (stronger focus on Android)
Doesn’t work in countries that don’t allow Google
User privacy concerns***

 

*** “Firebase has been claimed to be used by Google to track users without their knowledge. On July 14, 2020, a lawsuit was filed accusing Google of (…) logging what the users are looking at in many types of apps, despite the user following Google’s own instructions to turn off the web and app activity collected by the company.” (https://en.wikipedia.org/wiki/Firebase)anced settings.

Firebase Advantages ++ 

Cloud based
– Google: large team that supports and maintains it; very low risk of the company failing; however, Google has a reputation of discontinuing products / services, so there is no guarantee
– Backend as a service (ease of use)
– The Firebase Real-time Database has its own advantages:
– Pay as you go, price scales with usage
– Less iOS support (stronger focus on Android)

Firebase Disadvantages —

– Purely cloud based
Google: vendor lock-in (no migration tools prevents you from making your app portable), you cannot access your data as it is hosted on the Firebase server
– Less flexibility: You cannot optimize the backend to match your app’s needs
The Firebase Real-time Database has its own drawbacks
    –
the whole DB is a huge JSON file
    –
limited querying capabilities
    –
no way to efficiently filter data
    –
Easily disorganized, hard to navigate and search
– Cost insecurities, hard – impossible to predict
– Less iOS support (stronger focus on Android)
– Doesn’t work in the countries that don’t allow Google
User privacy concerns: “Firebase has been claimed to be used by Google to track users without their knowledge. On July 14, 2020, a lawsuit was filed accusing Google of (…) logging what the users are looking at in many types of apps, despite the user following Google’s own instructions to turn off the web and app activity collected by the company.” (https://en.wikipedia.org/wiki/Firebase)

Firebase alternatives: A look at out-of-the-box data sync solutions

The majority of offerings for developers that handle data synchronisation as defined here, are cloud-based and fall into the category of BaaS (can also be MBaaS (Mobile Backend as a Service) or PaaS (platform as a Service) or DBaaS (Database as a Service). Typically that measn that data synchronisation is only a small part of the offering. However, that is our sole focus here.

Data Sync Solution comparison matrix – Firebase and its alternatives

Solution name Company Category Data Sync IoT / Mobile Database Type of DB Cloud OS / Platforms Languages License
Cloudant Sync IBM (Cloudant was acquired in 2014) DBaaS
(Cloud DB and Cloud Sync)
Two-way
cloud data replication (called “sync”)
IoT
& Mobile
Cloud
database based on Couch DB
NoSQL;
distributed JSON document database
Cloud-based
replication to and from on-device data (CouchDB <> cloud service)
hosted
service
C#,
Java, JavaScript, Objective-C, PHP, Ruby
Proprietary
(CouchDB is Apache 2.0 and they integrate with several open source libraries)
Couchbase server and Couchbase Sync Gateway Couchbase (a merger of Couch One and Membase) Cloud
DB and Cloud Sync
Sync
needs a Couchbase Server
IoT
& Mobile
Edge:
Couchbase Lite; Server: Couchbase
NoSQL;
document database
Always
needs Couchbase Server (originally Membase)
mainly
used as hosted service;
iOS, Android, .NET (Desktop/Server), .NET UWP, Xamarin
Swift,
Objective-C, Java (Android), Java (Non-Android), Kotlin, C#, JavaScript, C
Apache
2.0, delayed open source
Firebase** Google (Firebase was acquired by Google in 2014) BaaS
(Cloud)
Cloud
Sync via Google servers
Mobile Cloud:
Firebase Realtime Database; Edge: Caching only (Firestore)
Document
store
hosted
only
APIS
for iOS & Android
JavaScript API
RESTful HTTP API
Java
JavaScript
Objective-C
proprietory
Mongo Realm Sync MongoDB
(Realm was acquired in 2019)
Cloud
DB and Cloud Sync
Sync
(in Alpha); only via Mongo Cloud
IoT
& Mobile
Cloud:
MongoDB, Edge: Mongo Realm
MongoDB:
NoSQL document store; RealmDB: Embedded NoSQL DB
hosted
service
MongoDB:
Linux, OS X, Solaris, Windows
Mongo Realm DB:
Android, iOS
20+
languages, e.g. Java, C, C#, C++
Mongo
DB changed its license from open source (GNU) to MongoDB Inc.’s Server Side
Public License (SSPL) in 2018.
ObjectBox
Sync
ObjectBox DB
and Sync

Offline
Sync, on-premise Sync, Cloud Sync

p2p Sync is planned

IoT
& Mobile
ObjectBox Object-oriented
embedded NoSQL DB
Self-hosted
/ on-premise; hosted service upon request only
iOS,
Android, Linux, Windows, MacOS, any POSIX-system
C,
C++
Java
Kotlin
Swift
Go
Flutter / Dart
Python
DB:
Open source bindings, Apache 2.0, proprietary core
Parse Originally
Parse, acquired by Facebook, closed down and open sourced, unmaintained
MBaaS
(Cloud)
Cloud
Sync, self-hosted or via a provider that offers Parse hosting
Mobile Both,
PostgreSQL* and MongoDB, can be used as a database for Parse
MongoDB:
NoSQL document store; PostgreSQL:
Only
Cloud, only self-hosted or via a provider that offers Parse hosting
Server: REST
API lets you interact with Parse Server from anything that can send an HTTP
request
open
source, BSD
Syncstudio HandApps Cloud-based
sync between SQLite and MS SQL Server based in the MS Sync Framework
Sync Mobile Edge:
SQLite or MSSQL (including LocalDB or Express); Server:
Microsoft SQL
relational
/ SQL
SQL
Server; Sync / replication works via cloud only
Android
Java, Basic4Android, Windows Forms, UWP, Windows Mobile, Xamarin
proprietory,
4 licenses available: Community/Free, Subscription, Perpetual and Royalty
Free
Zumero Zumero
LLC
Cloud-based
replication of SQL data for Mobile
Sync Mobile Edge:
SQLite; Server: Microsoft SQL
relational
/ SQL
SQL
Server; Sync / replication works via cloud only
Mobile
only (iOS, Android, Xamarin, PhoneGap)

proprietory,
annual license scaling with the number of devices

 

 

 

 

Notes: Microsoft Sync Framework (renamed Sync Framework Toolkit at some point) is a legacy open source product which MS no longer supports

* PostgreSQL vs Postgres
** There are many Cloud Sync alternatives to Firebase, we added the more prominent options and any service that also serves Edge Computing

Data Sync is no standardized term and though it seems to be in use by many big companies and most dvelopers will have a notion of what it is, the devil is in the details. So, we might have missed an important solution or taken an angle someone else would not agree with. Please feel free to let us know what to improve.

ObjectBox DB and Sync – designed to keep data up to date across time and space

ObjectBox is a high performance NoSQL fully ACID-compliant edge database built from scratch for efficient data on and across restricted and occasionally connected devices, taking care of keeping data in sync reliably. ObjectBox developer tools are easy to use, quick to implement, and optimized for high-performance and frugal resource-use on edge devices running mobile, desktop, server, and IoT applications. ObjectBox helps developers to focus on what they love and build great applications without needing to take care of the boilerplate code for resilient connectivity, synchronizing data, and tedious DB optimizations. This cuts down initial implementation efforts, ongoing maintenance efforts, undesired problems, and data loss – therefore reducing costs and time to market tremendously. We are dedicated to bring joy and delight to Mobile and IoT application developers.

What is Data Synchronization + How to Keep Data in Sync

What is Data Synchronization + How to Keep Data in Sync

What is Data Sync / Data Synchronization in app development?

Data Synchronization (Sync) is the process of establishing consistency and consolidation of data between different devices and fundamental to most IT solutions, especially in IoT and Mobile. Data Sync entails the continuous harmonization of data over time. It typically is a highly complex, non-trivial technical process. Even corporates struggle with its implementation and had to roll back synchronization solutions due to technical challenges. 

The question Data Sync answers is

phone-data-sync-with-machine-payment-automatic-data

How do you keep data sets from two (or more) data stores / databases – separated by space and time – mirrored with one another as closely as possible, in the most efficient way?

In solving this technical challenge one must deal with asynchrony, conflicts, slow bandwidth, flaky networks, third-party applications, and file systems that have different semantics.

Data Sync versus Data Replication in Databases

sync-data-better-than-replication

Data replication is the process of storing the same data in several locations to prevent data loss and improve data availability and accessibility. Typically, data replication means that all data is fully mirrored / backed up / replicated on another instance (device/server). This way, all data is stored at least twice. Replication typically works in one direction only (unidirectional); there is no additional logic to it and no possibility of conflicts.

In contrast, synchronization typically relates to a subset of the data (selection) and works in two directions (bi-directional), adding a layer of complexity, because now conflicts can arise. Of course, if you select all data for synchronisation into one direction, it will yield the same result as replication. However, replication cannot replace synchronization.

Why do you need to keep data in sync?

Think about it – if clocks were not in sync, everyone would live on a different time. While I can see an upside to this, it would result in many inefficiencies as you could not rely on schedules. When business data is not in sync (up-to-date everywhere), it harms the efficiency of the organization due to:

  • Isolated data silos
  • Conflicting data / information states
  • Duplicate data / double effort
  • Outdated information states / incorrect data

In the end, the members of such an organization would not be able to communicate and collaborate efficiently with each other. They would instead be spending a lot of time on unnecessary work and “conflict resolution”. On top, management would miss an accurate overview and data-driven insights to prioritize and steer the company. The underlying mechanism that keeps data up-to-date across devices is a technical process called data synchronization (Sync). And while we expect these processes to “just work”, someone needs to implement and maintain them, which is a non-trivial task.

Growing data masses and shifts in data privacy requirements call for sensible usage of network bandwidth and the cloud. Edge computing with selective data synchronization is an effective way to manage which data is sent to the cloud, and which data stays on the device. Keeping data on the edge and synchronizing selective data sets effectively, reduces the data volume that is transferred via the network and stored in the cloud. Accordingly, this means lower mobile networking and cloud costs. On top, it also enables higher data security and data privacy, because it makes it easy to store personal and private data with the user. When data stays with the user, data ownership is clear too.

Unidirectional Data Replication

replication-data-sync-database

Bidirectional Data Synchronization

how-to-sync-data-what-is-data-sync

Out-of-the-box Sync magic: Syncing is hard

Almost every mobile app or IoT application needs to sync data, so every developer is aware of the basic concept and challenges. This is why many experienced developers appreciate out-of-the-box solutions. While JSON / REST offers a great concept to transfer data, there is more to data Sync than what it looks like at a glance. Of course, the complexity of Sync varies widely depending on the use case. For example, the amount of data, data changes, synchronous / asynchronous sync, and number of devices (connections), and what kind of client-server or peer-to-peer setup is needed, all affect the complexity.

iceburg-building-data-synchronization

What looks easy in practice hides a complex bit of coding and opens a can of worms for testing. For an application to work seamlessly across devices – independent of the network, which can be offline, flaky, or only occasionally connected – an app developer must anticipate and handle a host of local and network failures to ensure data consistency. Moreover, for devices with restricted memory, battery and/or CPU resources (i.e. Mobile and IoT devices), resource sensitivity is also essential. Data storage and synchronization solutions must be both effective / efficient, and sustainable.

How to Keep Data in Sync Without the Headache?

Thankfully, there are out-of-the-box data synchronization solutions available on the market, which solve data syncing for developers. They fall broadly into two categories: cloud-dependent data synchronization, and independent, “edge” data synchronization. Cloud-based solutions, like Firebase, require a connection to the internet to function; data is synchronized to the back-end server to be stored. Edge solutions, like ObjectBox, can also operate offline; data is stored in an efficient on-device database, synchronization occurs only once the device goes online. There are only a few solutions to offer true edge synchronization, i.e. “peer to peer” data sync, directly between any two devices. Below, we summarize the most popular market offerings for data synchronization (offline and cloud based):

cloud-to-edge-data-sync-options-matrix-alternatives

Firebase

Firebase is a Backend as a Service (BaaS) offering from Google (acquired). Google offers it as a cloud hosted solution for mobile developers. 

Parse

Parse is a BaaS offering that Facebook acquired and shut down. Facebook open sourced the code. The GitHub repository is not officially maintained. You can host Parse yourself or use a Parse hosting service.

Mongo Realm

Realm was acquired by MongoDB in 2019; the Mongo Realm Sync solution is now in Alpha and available hosted with MongoDB.

Couchbase

Couchbase is a Cloud DB, Edge DB and Sync offering that requires the use of Couchbase servers.

ObjectBox

ObjectBox is a DB for any device, from restricted edge devices to servers, and offers an out-of-the-box Sync solution. ObjectBox enables self-hosting on-premise / in the cloud, as well as Offline Sync.

Data Sync, Edge Computing, and the Future of Data

Edge computing means storing and processing data close to the source of the data, i.e. on edge devices. It is a mega-shift happening in computing. Accordingly, the market is growing rapidly with projections estimating continuing growth with a 34% CAGR for the next five years. The move from the cloud to the edge is strongly driven by new use cases and growing data masses Edge data persistence and Data Sync, especially “Offline Sync”, are the key technologies needed for Edge Computing. Using edge data persistence, data can be stored and processed on the edge. This means the application always works, independent from a network connection, offline. Faster response times can be guaranteedWith Offline Sync, data can be synchronized between several edge devices, e.g. IoT gateways, smartphones, machines locally. Using Data Sync, selected data can be synchronized with  a central server, on premise or in the cloud. By exchanging less data with the cloud or a central instance, data synchronization reduces the burden on the network. This brings down mobile network and cloud costs, and reduces the amount of energy used: a win-win-win solution.

How Building Green IoT Solutions on the Edge Can Help Save Energy and CO2

How Building Green IoT Solutions on the Edge Can Help Save Energy and CO2

The internet of things (IoT) has a huge potential to reduce carbon emissions, as it enables new ways of operating, living, and working [1] that are more efficient and sustainable. However, IoT’s huge and growing electricity demands are a challenge. This demand is due primarily to the transmission and storage of data in cloud data centers. [2] While data center efficiency and the use of green energy will reduce the CO2 emissions needed for this practice, it is not addressing the problem directly. [3

iot-data-cloud-energy-waste

With ObjectBox, we address this unseen and fast-growing CO2 source at the root: ObjectBox empowers edge computing, reducing the volume of data transmitted to central data storage, while at the same time, heightening data transmission and storage efficiency. [4] We’ve talked before about how edge computing is necessary for a sustainable future, below we dive into the numbers a bit deeper. TLRD: ObjectBox enables companies to cut the power consumption of their IoT applications, and thus their emissions, by 50 – 90%. For 2025, the potential impact of ObjectBox is a carbon emission reduction of 594 million metric tons (see calculations below).

How ObjectBox’ Technology Reduces Overall Data Transmission

 ObjectBox reduces data transmission in two ways: 1. ObjectBox reduces the need for data transmission, 2. ObjectBox makes data transmission more efficient. ObjectBox’ database solution allows companies to build products that store and process data on edge devices and work with that data offline (as well as online). This

Green IoT Solution

not only improves performance and customer experience, it also reduces the overall volume of data that is being sent to the cloud, and thus the energy needed to transfer the data as well as store it in the cloud. ObjectBox’ Synchronization solution makes it easy for companies to transmit only the data that needs to be transmitted through 1) selective two-way syncing and 2) differential delta syncing. Synchronizing select data reduces the energy required for unnecessarily transmitting all data to the cloud.

We have demonstrated in exemplary case studies that ObjectBox can reduce total data transmissions by 70-90%, depending on the case. There will, however, typically be value in transmitting some parts of data to a central data center (cloud); ObjectBox Sync combines efficient compression based on standard and proprietary edge compression methods to keep this data small. ObjectBox also has very little overhead. Comparing the transmission of the same data sets, ObjectBox saves 40-60% on transmission data volume through the delta syncing and compression, and thus saves equivalent CO2 emissions for data transmissions. Additional studies support these results, and have shown that moving from a centralized to a distributed data structure, saves between 32 and 93% of transmission data. [5

sync-sustainable-data-save-energy

Calculations: How Does ObjectBox Save CO2?

Physically using a device consumes little energy directly; it is the wireless cloud infrastructure in the backend (data center storage and data transmission) that is responsible for the high carbon footprint of mobile phones [6] and IoT devices. Estimates say that IoT devices will produce around 2,8 ZB of data in 2020 (or 2,823,000,000,000  GB), globally. [7] Only a small portion of that data actually gets stored and used; we chose to use a conservative estimate of 5% [8] (141,150,000,000 GB) and of that portion, 90% is transferred to the cloud [9] (127,035,000,000 GB). Transferring 1 GB of data to the cloud and storing it there costs between 3 and 7 kWh. [10] Assuming an average of 5 kWh this means a 127,035,000,000 GB multiplied by 5kWh, resulting in a total energy expenditure of 635,175,000,000 kWh. Depending on the energy generation used, CO2 emissions vary. We are using a global average of 0,475 kgCO2 / 1 kwH. [11] In total this means that there will be 301,708,125,000 KG of CO2, or roughly 301 million metric tons of CO2 produced to transfer data to the cloud and store it there in 2020. 

Projections for 2025 have data volumes as high as 79.4 ZB. [12] Following the same calculations as above, IoT devices would be responsible for 8 billion metric tons of CO2 in 2025.* We estimate that using ObjectBox can cut CO2 caused by data transmission and data centers by 50-90%, by keeping the majority of data on the device, and transmitting data efficiently. It will take time for ObjectBox to enter the market, so assuming a 10% market saturation by 2025 and an average energy reduction of 70%, using ObjectBox could cut projected CO2 emissions by 594 million metric tons in 2025.

ObjectBox is on a mission to reduce digital waste which unnecessarily burdens bandwidth infrastructure and fills cloud servers, forcing the expansion of cloud farms and in turn, contributing to the pollution of the environment. As our digital world grows, we all need to give some thought to how we should structure our digital environments to optimize and support useful, beneficial solutions, while also keeping them efficient and sustainable. 

*Of course, in that time, the technologies will all be more efficient and thus use less electricity while at the same time CO2 emissions / kWh will have dropped too. Thus, we are aware that this projection is an oversimplification of a highly complex and constantly changing system.

[1] https://www.theclimategroup.org/sites/default/files/archive/files/Smart2020Report.pdf
[2] https://www.iea.org/reports/tracking-buildings/data-centres-and-data-transmission-networks
[3]“Data centres… have eaten into any progress we made to achieving Ireland’s 40% carbon emissions reduction target.” from https://www.climatechangenews.com/2017/12/11/tsunami-data-consume-one-fifth-global-electricity-2025/
[4] https://medium.com/stanford-magazine/carbon-and-the-cloud-d6f481b79dfe
[5] https://www.researchgate.net/publication/323867714_The_carbon_footprint_of_distributed_cloud_storage
[6] https://www.resilience.org/stories/2020-01-07/the-invisible-and-growing-ecological-footprint-of-digital-technology/
[7] https://www.idc.com/getdoc.jsp?containerId=prUS45213219, https://priceonomics.com/the-iot-data-explosion-how-big-is-the-iot-data/, https://www.gartner.com/en/newsroom/press-releases/2018-11-07-gartner-identifies-top-10-strategic-iot-technologies-and-trends, https://www.iotjournaal.nl/wp-content/uploads/2017/02/white-paper-c11-738085.pdf, ObjectBox research
[8] Forrester (https://internetofthingsagenda.techtarget.com/blog/IoT-Agenda/Preventing-IoT-data-waste-with-the-intelligent-edge), Harvard BR (https://hbr.org/2017/05/whats-your-data-strategy), IBM (http://www.redbooks.ibm.com/redbooks/pdfs/sg248435.pdf), McKinsey (https://www.mckinsey.com/business-functions/mckinsey-digital/our-insights/the-internet-of-things-the-value-of-digitizing-the-physical-world)
[9] https://www.gartner.com/smarterwithgartner/what-edge-computing-means-for-infrastructure-and-operations-leaders/
[10] According to the American Council for an Energy-Efficient Economy: 5,12 kWh of electricity / GB of transferred data. According to a Carnegie Mellon University study: 7 kWh / GB. The American Council for an Energy-Efficient Economy concluded: 3.1 kWh / GB.
[11] https://www.iea.org/reports/global-energy-co2-status-report-2019/emissions
[12] https://www.idc.com/getdoc.jsp?containerId=prUS45213219

Why do we need Edge Computing for a sustainable future?

Why do we need Edge Computing for a sustainable future?

Centralized data centers consume a lot of energy, produce a lot of carbon emissions and cause significant electronic waste. While data centers are seeing a positive trend towards using green energy, an even more sustainable approach (alongside so-called “green data centers” [1]) is to cut unnecessary cloud traffic, central computation and storage as much as possible by shifting computation to the edge. Ideally, Edge Computing strategies use efficient technologies like ObjectBox to harness the power of already deployed available devices (like e.g. smartphones, machines, desktops, gateways), making the solution even more sustainable.

Why do Digitisation and IoT projects need to think about sustainability now?

Huge centralized data centres (cloud computing) have become a critical part of the infrastructure for a digitalized society. These large central cloud data centers produce a lot of carbon emissions, electric and electronic waste. [2] The share of global electricity used by data centres is already estimated to be around 1-3% [3] and data centers generate 2% of worldwide CO2 emissions (on par with the aviation industry). [4]

54% of which are caused by the cloud data centers of the big hyperscalers (Google, Amazon, Microsoft, Alibaba Cloud). [5] On top of this, providing and maintaining cloud infrastructure (manufacturing, shipping of hardware, buildings and lines) also consumes a huge amount of greenhouse gases [3] and produces a lot of abnormal waste (e.g. toxic coolants) at the end of life. [6]

sustainable edge computing

Bearing that in mind, the growth forecasts for digitization, IoT, and Mobile [7] are concerning. The steady increase in data processing, storage, and traffic in the future, comes with a huge electricity demand for this industry. [8] In fact, estimations expect the communications industry to use 20% of all the world’s electricity by 2025. [9]

sustainable edge computing

Shifting to green energy is a good step. However, a more effective and ultimately longer term solution requires looking at the current model of data storage, filtering, processing and transferal. By implementing Edge Computing, we can reduce the amount of useless and wasteful data traversing to and from the cloud as much as possible, thus reducing overall energy requirements in the long term.

What is Edge Computing?

While until recently 90 percent of enterprise data was sent to the cloud, this is changing rapidly. In fact, this number is dropping to only 25 percent in the next 3 years according to Gartner. By then, most of the data will be stored and used locally, on the device it was created on, e.g. on smartphones, cars, trains, machines, watches. This is called Edge Computing. Accordingly, edge devices need the same technology stack (just in a much smaller format) as a cloud server. This means: An operating system, a data storage / persistence layer (database), a networking layer, security functionalities etc. that run efficiently on restricted hardware.

As you can only use the devices’ resources, which can be pretty limited, inefficient applications can push a device to its limits, leading to slow response rates, crashes, and battery drain.

edge device architecture

EDGE DEVICE ARCHITECTURE

Edge Computing is much more than some simple data pre-processing, which takes advantage of only a small portion of the computing that is possible on the edge. An on-device database is a prerequisite for meaningful Edge Computing. With an on-device database, data can be stored and processed on the devices directly (the so called edge). Only useful data is sent to the server and saved there, reducing the networking traffic and computing power used in data centers tremendously, while also making use of the computing resources of devices which are already in use. This greatly reduces bandwidth and energy required by data centers. On top, edge computing also provides the flexibility to operate independent from an Internet connection, enables fast real time response rates, and cuts cloud costs.

Why is Edge Computing sustainable?

Edge Computing reduces network traffic and cloud data center usage

With Edge Computing the amount of data traversing the network can be reduced greatly, freeing up bandwidth. Bandwidth refers to the transmission speed of data on the network. While maximum speeds are published by the network operators, the actual speed obtained in a given network is almost always lower, because the bandwidth is shared and limited. The more data transferred at any given moment, the slower the network speed. Data on the edge is also much more likely to be used, and then (due to restricted devices size) deleted when it is no longer useful.

Edge computing is optimized for efficiency

Edge “data centres” are typically more efficient than cloud data centres. As described above, resources on edge devices are restricted – as opposed to cloud infrastructure, edge devices do not scale horizontally. Therefore, every piece of the tech stack is – ideally – highly optimized for resource efficiency. Any computing done more efficiently helps reduce energy consumption, especially taking into account the huge number of devices already deployed (number), the worldwide impact is significant.

With Edge Computing you can put already deployed hardware to better use

On top, there is a realm of edge devices already deployed that is currently underused. Many existing devices are capable of fairly complex computing; when these devices send all of their data to the cloud, an opportunity is lost. Edge Computing utilizes existing hardware and infrastructure,  taking advantage of the existing computing power. If these devices continue to be underused, we will need to build bigger and bigger central data centers, simultaneously burdening existing network infrastructure and reducing bandwidth for senselessly sending everything to the cloud.

Cloud versus Edge: an Example

Today, many projects are built based on cloud computing. Especially in first prototypes or pilots, cloud computing offers an easy and fast start. However, with scale, cloud computing often becomes too slow, expensive, and unreliable. In a typical cloud setup, data is gathered on edge devices and forwarded to the cloud for computation and storage. Often a computed result is sent back. In this design, the edge devices are dumb devices that are dependant upon a working internet connection and a working cloud server; they do not have any intelligence or logic of their own. In a smart home cloud example, data would be sent from devices in the home, e.g. a thermostat, the door, the TV etc. to the cloud, where it is saved and used.

Cloud vs Edge

If the user would want to make changes via a cloud-based mobile app when in the house, the changes would be send to the cloud, changed there and then from there be sent to the devices. When the Internet connection is down or the server is not working, the application will not work.

With Edge Computing, data stays where it is produced, used and where it belongs – without traversing the network unnecessarily. This way, cloud infrastructure needs are reduced in three ways: Firstly, less network traffic, secondly, less central storage and thirdly less computational power. Rather, edge computing makes use of all the capable hardware already deployed in the world. E.g. in a smart home, all the data could stay within the house and be used on site. Only the small part of the data truly needed accessible from anywhere would be synchronized to the cloud.

Cloud vs Edge

Take for example a thermostat in such a home setting: it might produce 1000s of temperature data points per minute. However, minimal changes typically do not matter and data updates aren’t necessary every millisecond. On top, you really do not need all this data in the cloud and accessible from anywhere.

With Edge Computing, this data can stay on the edge and be used within the smart home as needed. Edge Computing enables the smart home to work fast, efficiently, and autonomous from a working internet connection. In addition, the smart home owner can keep the private data to him/herself and is less vulnerable to hacker attacks. 

How does ObjectBox make Edge Computing even more sustainable?

ObjectBox improves the sustainability of Edge Computing with high performance and efficiency: our 10X speed advantage translates into less use of CPU and battery / electricity. With ObjectBox, devices compute 10 times as much data with equivalent power. Due to the small size and efficiency, ObjectBox runs on restricted devices allowing application developers to utilize existing hardware longer and/or to do more instead on existing infrastructure / hardware.

Alongside the performance and size advantages, ObjectBox’ Sync solution takes care of making data available where needed when needed. It allows synchronization in an offline setting and / or to the cloud. Based on efficient syncing principles, ObjectBox Sync aims to reduce unnecessary data traffic as much as possible and is therefore perfectly suited for efficient, useful, and sustainable Edge Computing. Even when syncing the same amount of data, ObjectBox Sync reduces the bandwidth needed and thus cloud networking usage, which incidentally reduces cloud costs.

ObjectBox’ Time Series feature, provides users an intuitive dashboard to see patterns behind the data, further helping users to track thousands of data points/second in real-time.

How Edge Computing enables new use cases that help make the world more sustainable

As mentioned above, there are a variety of IoT applications that help reduce waste of all kinds. These applications can have a huge impact on creating a more sustainable world, assuming the applications themselves are sustainable. Three powerful examples to demonstrate the huge impact IoT applications can have on the world:

1) Smart City Lighting: Chicago has implemented a system which allows them to save approx. 10 million USD / year and London estimates it can save up to 70% of current electricity use and costs as well as maintenance costs through smart public lighting systems. [10]

2) Reducing Food Waste: From farm to kitchen, IoT applications can help to reduce food waste across the food chain. Sensors used to monitor the cold chain, from field to supermarket, can ensure that food maintains a certain temperature, thus guaranteeing that products remain food safe and fresh longer, reducing food waste.

3) Reduce Water Waste: Many homes and commercial building landscapes are still watered manually or on a set schedule. This is an inexact method of watering, which does not take into account weather, soil moistness, or the water levels needed by the plant. Using smart IoT water management solution, landscape irrigation can be reduced, saving water and improving landscape health.

These positive effects are all the more powerful when the IoT applications themselves are sustainable. 

The benefits of cloud computing are broad and powerful, however there are costs to this technology. A combination of green data centers and Edge Computing helps to resolve these often unseen costs. With Edge Computing we can reduce the unnecessary use of bandwidth and server capacity (which comes down to infrastructure, electricity and physical space) while simultaneously taking advantage of underused device resources. ObjectBox amplifies these benefits, with high performance on small devices and efficient data synchronization – making edge computing an even more sustainable solution.