What is Data Synchronization + How to Keep Data in Sync

What is Data Synchronization + How to Keep Data in Sync

What is Data Sync / Data Synchronization in app development?

Data Synchronization (Sync) is the process of establishing consistency and consolidation of data between different devices and fundamental to most IT solutions, especially in IoT and Mobile. Data Sync entails the continuous harmonization of data over time. It typically is a highly complex, non-trivial technical process. Even corporates struggle with its implementation and had to roll back synchronization solutions due to technical challenges. 

The question Data Sync answers is

phone-data-sync-with-machine-payment-automatic-data

How do you keep data sets from two (or more) data stores / databases – separated by space and time – mirrored with one another as closely as possible, in the most efficient way?

In solving this technical challenge one must deal with asynchrony, conflicts, slow bandwidth, flaky networks, third-party applications, and file systems that have different semantics.

Data Sync versus Data Replication in Databases

sync-data-better-than-replication

Data replication is the process of storing the same data in several locations to prevent data loss and improve data availability and accessibility. Typically, data replication means that all data is fully mirrored / backed up / replicated on another instance (device/server). This way, all data is stored at least twice. Replication typically works in one direction only (unidirectional); there is no additional logic to it and no possibility of conflicts.

In contrast, synchronization typically relates to a subset of the data (selection) and works in two directions (bi-directional), adding a layer of complexity, because now conflicts can arise. Of course, if you select all data for synchronisation into one direction, it will yield the same result as replication. However, replication cannot replace synchronization.

Why do you need to keep data in sync?

Think about it – if clocks were not in sync, everyone would live on a different time. While I can see an upside to this, it would result in many inefficiencies as you could not rely on schedules. When business data is not in sync (up-to-date everywhere), it harms the efficiency of the organization due to:

  • Isolated data silos
  • Conflicting data / information states
  • Duplicate data / double effort
  • Outdated information states / incorrect data

In the end, the members of such an organization would not be able to communicate and collaborate efficiently with each other. They would instead be spending a lot of time on unnecessary work and “conflict resolution”. On top, management would miss an accurate overview and data-driven insights to prioritize and steer the company. The underlying mechanism that keeps data up-to-date across devices is a technical process called data synchronization (Sync). And while we expect these processes to “just work”, someone needs to implement and maintain them, which is a non-trivial task.

Growing data masses and shifts in data privacy requirements call for sensible usage of network bandwidth and the cloud. Edge computing with selective data synchronization is an effective way to manage which data is sent to the cloud, and which data stays on the device. Keeping data on the edge and synchronizing selective data sets effectively, reduces the data volume that is transferred via the network and stored in the cloud. Accordingly, this means lower mobile networking and cloud costs. On top, it also enables higher data security and data privacy, because it makes it easy to store personal and private data with the user. When data stays with the user, data ownership is clear too.

Unidirectional Data Replication

replication-data-sync-database

Bidirectional Data Synchronization

how-to-sync-data-what-is-data-sync

Out-of-the-box Sync magic: Syncing is hard

Almost every mobile app or IoT application needs to sync data, so every developer is aware of the basic concept and challenges. This is why many experienced developers appreciate out-of-the-box solutions. While JSON / REST offers a great concept to transfer data, there is more to data Sync than what it looks like at a glance. Of course, the complexity of Sync varies widely depending on the use case. For example, the amount of data, data changes, synchronous / asynchronous sync, and number of devices (connections), and what kind of client-server or peer-to-peer setup is needed, all affect the complexity.

iceburg-building-data-synchronization

What looks easy in practice hides a complex bit of coding and opens a can of worms for testing. For an application to work seamlessly across devices – independent of the network, which can be offline, flaky, or only occasionally connected – an app developer must anticipate and handle a host of local and network failures to ensure data consistency. Moreover, for devices with restricted memory, battery and/or CPU resources (i.e. Mobile and IoT devices), resource sensitivity is also essential. Data storage and synchronization solutions must be both effective / efficient, and sustainable.

How to Keep Data in Sync Without the Headache?

Thankfully, there are out-of-the-box data synchronization solutions available on the market, which solve data syncing for developers. They fall broadly into two categories: cloud-dependent data synchronization, and independent, “edge” data synchronization. Cloud-based solutions, like Firebase, require a connection to the internet to function; data is synchronized to the back-end server to be stored. Edge solutions, like ObjectBox, can also operate offline; data is stored in an efficient on-device database, synchronization occurs only once the device goes online. There are only a few solutions to offer true edge synchronization, i.e. “peer to peer” data sync, directly between any two devices. Below, we summarize the most popular market offerings for data synchronization (offline and cloud based):

cloud-to-edge-data-sync-options-matrix-alternatives

Firebase

Firebase is a Backend as a Service (BaaS) offering from Google (acquired). Google offers it as a cloud hosted solution for mobile developers. 

Parse

Parse is a BaaS offering that Facebook acquired and shut down. Facebook open sourced the code. The GitHub repository is not officially maintained. You can host Parse yourself or use a Parse hosting service.

Mongo Realm

Realm was acquired by MongoDB in 2019; the Mongo Realm Sync solution is now in Alpha and available hosted with MongoDB.

Couchbase

Couchbase is a Cloud DB, Edge DB and Sync offering that requires the use of Couchbase servers.

ObjectBox

ObjectBox is a DB for any device, from restricted edge devices to servers, and offers an out-of-the-box Sync solution. ObjectBox enables self-hosting on-premise / in the cloud, as well as Offline Sync.

Data Sync, Edge Computing, and the Future of Data

Edge computing means storing and processing data close to the source of the data, i.e. on edge devices. It is a mega-shift happening in computing. Accordingly, the market is growing rapidly with projections estimating continuing growth with a 34% CAGR for the next five years. The move from the cloud to the edge is strongly driven by new use cases and growing data masses Edge data persistence and Data Sync, especially “Offline Sync”, are the key technologies needed for Edge Computing. Using edge data persistence, data can be stored and processed on the edge. This means the application always works, independent from a network connection, offline. Faster response times can be guaranteedWith Offline Sync, data can be synchronized between several edge devices, e.g. IoT gateways, smartphones, machines locally. Using Data Sync, selected data can be synchronized with  a central server, on premise or in the cloud. By exchanging less data with the cloud or a central instance, data synchronization reduces the burden on the network. This brings down mobile network and cloud costs, and reduces the amount of energy used: a win-win-win solution.

Why Edge Computing is More Relevant in 2021 Than Ever

Why Edge Computing is More Relevant in 2021 Than Ever

The world has been forced to digitize more quickly and to a greater extent in 2020 and 2021. COVID has created the need to remodel how work, socializing, production, entertainment, and supply chains function. Despite decades of digitization efforts, with the pandemic upon us, digitization challenges have become transparent. Many companies and countries realize now, they have fallen behind. And those that have not yet digitized were hit hardest by the pandemic. [1] With people leaning heavily on online digital solutions, internet infrastructure is at its capacity limit. [2] Accordingly, users are seeing broadband speeds drop by as much as half. [3] In Europe, governments even requested to reduce the quality of Netflix, Amazon Prime, Youtube and other streaming services to improve network speed. [4]

These challenges demonstrate the growing need for an alternative to cloud computing. Cloud computing is an inherently centralized computing paradigm. Edge Computing helps overcome many of the disadvantages of centralized computing. Edge Computing is inherently decentralized and keeps data local, at the ‘edge’ of the network. Edge Computing is ideal for both, data-intensive content and latency-sensitive applications. Edge Computing makes efficient use of local data and reduces the amount of traffic in the network.

edge computing 2020

Coronavirus accelerates the need to digitize

It was clear even before the outbreak that internet infrastructure was struggling to keep up with growing data volumes. However, the pandemic has made broadband limitations more apparent to everyday users.

Projections estimate that by 2025 there will be 20 million IoT devices [5] and 1.7MB of data created per second per person. It is slow, expensive, and wasteful to send all of this data to the cloud for storage and processing. This practice overburdens bandwidth and data center infrastructure. It makes projects expensive and unsustainable. Working with the data, locally, on the edge, where it was produced and is used, is more efficient than sending everything to the cloud and back. It brings reduced latency, reduced cloud usage and costs, independence from a network connection, more secure data and heightened data privacy – and even reduces CO2. Indeed, prior to the pandemic, edge computing was on the strategic roadmap for over 50% of mobility decision makers. [6]

As the world begins to recover from the coronavirus pandemic, digitization efforts will no doubt increase. We will see intelligent systems implemented across industries and value chains, accelerating innovation and alongside: data volumes and subsequent strain on network bandwidth. Edge computing is a key technology to ensure that this digitalization is both scalable and sustainable.  

Edge Computing takes the ‘edge’ off bandwidth strain

what is edge computing?

What is Edge Computing?

With edge computing, data is stored and used on devices at the “edge” of the network – away from centralized cloud servers. Computing on the edge means that data is stored and used locally, on the device, e.g. a smart phone or IoT device. Edge computing delivers faster decision making, local and offline data processing, as well as reduced data transfer to the cloud (e.g. filtered, computed, extra- or interpolated data), which saves both bandwidth and cloud storage costs. 

The Edge complements the Cloud

Although some might set cloud and edge in competition, the reality is that edge computing and cloud computing are both useful and relevant technologies. Both have different strengths and ideal use cases. Together they can provide the best of both worlds: decentralized local storage and processing, making efficient use of hardware on the edge and central storing and processing of some data, enabling additional centralized insights, data backups (redundancy), and remote access. To combine the best of both worlds, relevant and useful data must be synchronized between the edge and cloud in a smart and efficient way.  

Edge computing is an ideal technology to reduce the strain on data centers, so those functions that need cloud connection have adequate bandwidth; while those use cases that benefit from reduced latency and offline functionality are optimized on the edge.

The Edge: interface between the Physical and the Digital World

Edge devices handle the interface between the physical world and the cloud, enabling a whole set of new use cases. “Data-driven experiences are rich, immersive and immediate. But they’re also delay-intolerant data hogs”. [8] And therefore need to happen locally, on the edge. We may see edge computing enabling new forms of remote engagement [9], particularly in a post-corona environment.

Edge devices can be anything from a thermostat or small sensor to a fridge or mobile phone or car – and they are part of our direct physical world and use data from their local environment to enable new use cases. Think self-stocking fridges, self-driving cars, drone-delivered pizzas. In the same way, Edge Computing is the key to the first real world search engine. I am waiting for it every day: “Hey Google, where are my keys?” Within a location like a house, the concepts and technologies to enable such a real-world search engine are all clear and available – it is just a matter of time and ongoing digitization. The basis will need to be a fast and sustainable edge infrastructure. 

Sustainability on the Edge

Centralized data centers consume a lot of energy, produce a lot of carbon emissions and cause significant electronic waste. [10] While data centers are seeing a positive trend towards using green data centers, an even more sustainable approach is to cut unnecessary cloud traffic, central computation and storage as much as possible by shifting computation to the edge. Edge Computing strategies that harness the power of already deployed available hardware (like e.g. smartphones, machines, desktops, gateways) make the solution even more sustainable.

sustainability on the edge

Intelligent Edge: AI and Edge advance hand in hand

The growth of Artificial Intelligence (AI) and the Edge will go hand in hand. As more and more data is generated at the edge of the network, there will be a greater demand for intelligent data processing and structured optimization to reduce raw data loads going to the cloud. [11] Edge AI will have the power to work with data on local devices, keeping data streams more useful and usable. In the near future, Machine Learning applications will have the ability to learn and create unique, localized, decentralized insights on the edge – based on local inputs.

“With Edge AI, personalization features that we want from the app can be achieved on device. Transferring data over networks and into cloud-based servers allows for latency. At each endpoint, there are security risks involved in the data transfer”. [12] Which is part of the reason why the Edge AI Software market is forecasted to reach 1.12 trillion dollars volume by 2023. The development of AI accelerators, which improve model inferencing on the edge, namely from NVIDIA, Intel and Google are helping to make AI on the edge more viable. [13] A fast edge database is a necessary base technology to enable more AI on the edge. 

Edge Computing – an answer to Data Privacy concerns and a need for Resilience

As data collection grows in both breadth and depth, there is a stronger need for data privacy and security. Edge computing is one way to tackle this challenge: keeping data where it is produced, locally, makes data ownership clear and data less likely to be attacked and compromised. If compromised, the data compromised is clearly defined, making notification and subsequent actions manageable. ObjectBox, in its core and as an edge technology, is designed to keep data private, on those devices it was created on, and only share select data as needed. 

The more our private and working lives as well as the larger economy depend on digitalization, the more important it is that systems, underlying computing paradigms as well as networks have strong resilience and security. In computer networking, resilience is the ability to “provide and maintain an acceptable level of service in the face of faults and challenges to normal operation.” [14]

ing initEdge Computing shifts computer workloads – the collection, processing, and storage of data – from central locations (like the cloud) to the edge of the networks to many individual devices such as cell phones. Accordingly, any strain is distributed to many devices. Therefore, the risk of a total breakdown is reduced: If one device does not work anymore, the rest is still working. Depending on the setup, the individual devices could even compensate for devices that have a problem.

The same applies to security risks: Even if data from one device is compromised, all other data sets are still safe; the loss is thus very limited and clear.  Overall, as a complement to the cloud, edge computing provides improved strength and security in local networks around the world. These local infrastructures can relieve the pressure on the existing complex dependencies, and in turn make the wider system more resilient and flexible. With Edge Computing crisis response can therefore in all likelihood be faster, better informed, and more effective. [15]

Why Corona-Tracking-Apps need to work on the edge

There was initially quite some debate about taking a centralized versus decentralized approach to Corona-Tracking-Apps. [16] Many people were worried about their data. Edge Computing – storing most parts of the data locally, on the user’s device – is a great way to avoid unnecessary data sharing and keep data ownership clear. At the same time, data is by and large much more secure and less likely to be attacked and hacked, as the data to be gained is very reduced. An intelligent syncing mechanism like ObjectBox Sync ensures that the data which needs to be shared, is shared in a selective, transparent and secure way.

The next few years will see big cultural changes in both our personal and professional lives – a portion of those changes will be driven by increased digitalization. Edge computing is an important paradigm to ensure these changes are sustainable, scalable, and secure. Ultimately, we have the chance to rise from this crisis with new insights, new innovation, and a more sustainable future.

1. https://www.netzoekonom.de/2020/04/11/die-oekonomie-nach-corona-digitalisierung-und-automatisierung-in-hoechstgeschwindigkeit/
2. https://www.cnet.com/news/coronavirus-has-made-peak-internet-usage-into-the-new-normal/
3. https://www.nytimes.com/2020/03/26/business/coronavirus-internet-traffic-speed.html
4. https://www.theverge.com/2020/3/27/21195358/streaming-netflix-disney-hbo-now-youtube-twitch-amazon-prime-video-coronavirus-broadband-network
5. https://www.gartner.com/imagesrv/books/iot/iotEbook_digital.pdf
6. https://www.forbes.com/sites/forrester/2019/12/02/predictions-2020-edge-computing-makes-the-leap/#1aba50104201
7. https://www.gartner.com/smarterwithgartner/what-edge-computing-means-for-infrastructure-and-operations-leaders/
8. https://www.iotworldtoday.com/2020/03/19/ai-at-the-edge-still-mostly-consumer-not-enterprise-market/
9. https://www.accenture.com/us-en/insights/high-tech/edge-processing-remote-viewership
10. https://link.springer.com/article/10.1007/s12053-019-09833-8
11. https://www.forbes.com/sites/cognitiveworld/2020/04/16/edge-ai-is-the-future-intel-and-udacity-are-teaming-up-to-train-developers/#232c8fab68f2
12. https://www.forbes.com/sites/cognitiveworld/2020/04/16/edge-ai-is-the-future-intel-and-udacity-are-teaming-up-to-train-developers/#232c8fab68f2
13. https://www.forbes.com/sites/janakirammsv/2019/07/15/how-ai-accelerators-are-changing-the-face-of-edge-computing/#2c1304ce674f
14. https://en.wikipedia.org/wiki/Resilience_(network)
15. https://www.coindesk.com/how-edge-computing-can-make-us-more-resilient-in-a-crisis
16. https://venturebeat.com/2020/04/13/what-privacy-preserving-coronavirus-tracing-apps-need-to-succeed/

Digital Healthcare – a look at the market, projections, and trends with in-depth white paper

Digital Healthcare – a look at the market, projections, and trends with in-depth white paper

If you work in the healthcare industry, you are likely familiar with some uses of IoT devices. According to Gartner (2020), 79% of healthcare providers are already successfully employing IoT solutions.[1] However, this is just the beginning. While before COVID-19, the growth of digital health adoption had stalled [2], the market is picking up speed again. Indeed, Q3 2020 was a record year for investments in healthcare companies [3] and the market expects rising investments in healthtech for next years [4]. Today, underutilized data plays a major role in healthtech innovation [17] and the growing importance of healthcare data for future offerings is evident [5]. Take a look how analyts from Gartner to Accenture and Forrester expect the market to grow:

The digital healthcare market 2020 and beyond

digital-healthcare-market-trends-2020-edge-iot
  • Analysts expect Artificial Intelligence in healthcare to reach $6.6 billion by 2021 (with a 40% CAGR). [6]
  • The Internet of Medical Things (IoMT) market is expected to cross $136 billion by 2021. [11
  • Analysts expect the healthcare wearable market to have a market volume of $27 billion by 2023 (with a 27.9% CAGR). [7]
  • The IoT industry is projected to be worth $6.2 trillion by 2025 and around 30% of that market (or about $167 billion) will come from healthcare. [8]
  • Analysts expect the global Medical Health Apps market to grow to $236 billion by 2026, reflecting a shift towards value based care. [9]
  • The projected global digital health market is estimated to reach $510.4 billion by 2026 (with a 29% CAGR). [10]

The Healthcare industry has been struggling with shrinking payments and cost optimizations for years. [18] Fueled by the need to adopt in light of the COVID pandemic, digital technologies bring extensive changes quickly to this struggling industry now. Data is moving to the center of this changing ecosystem and harbors both risks and opportunities in a new dimension. [21] The basic architecture and infrastructure to have the data reliably, securely and quickly available where they are needed will be decisive for the success or failure of digital healthcare solutions. [17] [21]

We recommend keeping an eye on the following five trends

The 5 biggest digital healthcare trends to watch

AI-health-growth-market-tech

Artificial Intelligence (AI)  

Accenture estimates that AI applications can help save up to $150 billion annually for the US healthcare economy by 2026. [6] Therefore, it is no wonder that the healthcare sector is expected to be among the top five industries investing in AI in the next couple of years. [19] The top three greatest near-term value AI applications in healthcare are: 1. robot-assisted surgery ($40 billion), 2. virtual nursing assistants ($20 billion), and 3. administrative workflow assistance ($18 billion). 

big-data-health-analytics

Big Data / Analytics

The goal of big data analytic solutions is to improve the quality of patient care and the overall healthcare ecosystem. The global healthcare Big Data Analytics market is predicted to reach $39 billion by 2025. [12] The main areas of growth are medical data generation in the form of Electronic Health Records (EHR), biometric data, sensors data. 

internet-of-medical-things-digital-healthtech

Internet of Medical Things (IoMT)

IoMT is expected to grow to $508.8 billion by 2027. [13] According to Gartner, 79% of healthcare providers are already using IoT in their processes. [27] During COVID, IoMT devices have been used to increase safety and efficiency in healthcare, i.e. providing and automating clinical assistance and treatment to the infected patient, to lessen the burden of specialists. Future applications, like augmented reality glasses that assist during surgery, are leading to a focus more on IoMT-centric investments. [14]

telemedicine-virtual-healthcare-online

Telehealth / Telemedicine

Telecommunications technology enables doctors to diagnose and treat patients remotely. Consumer adoption of telehealth has skyrocketed in 2020 and McKinsey believes that up to $250 billion of current US healthcare spend could potentially be virtualized. [25] Also, many patients view telehealth offerings more favorable and – having made good experiences – are planning to continue using telehealth in the future. [26] Not astonishingly, telemedicine stocks also grow rapidly. [14]

edge-computing-hospital-clinic-offline

Edge Computing

Edge computing is a technological megashift happening in computing. [23] Instead of pushing data to the cloud to be computed, processing is done locally, on ‘the edge’. [15] Edge Computing is one of the key technologies to make healthcare more connected, secure, and efficient. [22]  Indeed, the digital healthcare ecosystem of the future depends on an infrastructure layer that makes health data accessible when needed where needed (data liquidity). [21] Accordingly, IDC expects the worldwide edge computing market to reach $250.6 billion in 2024 with a (12.5% CAGR) [24with healthcare identified as one of the leading industries that will adopt edge computing. [16

In-depth market overview with a look at the major market challenges and in-depth use cases

The healthcare market is in the middle of a fast digital transformation process. Drivers such as COVID,  growing IoT adoption in healthcare, and underlying social mega-trends are pushing digital healthcare growth to new heights. Therefore, the digital healthcare industry faces many challenges, both technical and regulatory. At the same time the healthcare market is offered a wealth of opportunities.

“While the challenges are numerous, leaders who seize the mindset that “disruptive change provides an opportunity to separate yourself from the pack” will build organizations meaningfully stronger than the ones they ran going into the crisis.” [20]

Interested in getting an in-depth fresh assessment of the digital healthcare landscape, including market drivers, biggest challenges and detailed use cases? 

Access your copy of the Digital Healthcare in 2020: Digitalization, IoT and Edge Computing white paper by singing up below:

By submitting this form, you agree to ObjectBox’ privacy policies.

Digital Healthcare whitepaper, table of contents

The Digital Healthcare Market 2020 and Beyond, a High Speed Revolution

Drivers of Growth and Change in Digital Healthcare

– COVID
– Growing Medical IoT Device Adoption
– The Explosion of Health Data
– Technological innovations: Edge Computing, AI, and VR
– Underlying Social Megatrends

The Challenges of Digital Healthcare / Healthtech

– Data Volume and Availability
– Reliability, Privacy, and Data Security
– IT Costs

Why Edge Computing is Integral to Data-driven Healthcare Ecosystems

– A quick look at Cloud and Edge Computing
– How Edge Computing contributes to resilient and secure healthcare data systems
– Transforming Healthcare with Edge Computing
– Use Case: Secure and Up to Date Digital Record Keeping in Doctors Offices
– Use Case: Connected Ambulances – Real Time Data from Home to Hospital

 

Digital Healthcare: Key Take-Aways

References

[1] https://www.computerworld.com/article/3529427/how-iot-is-becoming-the-pulse-of-healthcare.html / https://www.gartner.com/en/documents/3970072
[2] https://www.accenture.com/us-en/insights/health/leaders-make-recent-digital-health-gains-last
[3] https://sifted.eu/articles/europes-healthtech-industry-2020/
[4] https://www.mobihealthnews.com/news/emea/health-tech-investments-will-continue-rise-2020-according-silicon-valley-bank
[5] https://news.crunchbase.com/news/for-health-tech-startups-data-is-their-lifeline-now-more-than-ever/
[6] https://www.accenture.com/us-en/insight-artificial-intelligence-healthcare%C2%A0
[7] https://www.grandviewresearch.com/industry-analysis/wearable-medical-devices-market
[8] https://www.marketsandmarkets.com/PressReleases/iot-healthcare.asp
[9] https://www.grandviewresearch.com/press-release/global-mhealth-app-market
[10] https://www.globenewswire.com/news-release/2020/05/23/2037920/0/en/Global-Digital-Health-Market-was-Valued-at-USD-111-4-billion-in-2019-and-is-Expected-to-Reach-USD-510-4-billion-by-2025-Observing-a-CAGR-of-29-0-during-2020-2025-VynZ-Research.html
[11] https://www2.stardust-testing.com/en/the-digital-transformation-trends-and-challenges-in-healthcare
[12] https://www.prnewswire.com/news-releases/healthcare-analytics-market-size-to-reach-usd-40-781-billion-by-2025–cagr-of-23-55—valuates-reports-301041851.html#:~:text=Healthcare%20Big%20Data%20Analytics%20Market,13.6%25%20during%202019%2D2025 
[13] https://www.globenewswire.com/news-release/2020/11/25/2133473/0/en/Global-Digital-Health-Market-Report-2020-Market-is-Expected-to-Witness-a-37-1-Spike-in-Growth-in-2021-and-will-Continue-to-Grow-and-Reach-US-508-8-Billion-by-2027.html
[14] https://www.nasdaq.com/articles/iomt-meets-new-healthcare-needs%3A-3-medtech-trends-to-watch-2020-11-27
[15] https://go.forrester.com/blogs/predictions-2021-technology-diversity-drives-iot-growth/
[16] https://www.prnewswire.com/news-releases/state-of-the-edge-forecasts-edge-computing-infrastructure-marketworth-700-billion-by-2028-300969120.html
[17] https://news.crunchbase.com/news/for-health-tech-startups-data-is-their-lifeline-now-more-than-ever/ 
[18] https://www.gartner.com/en/newsroom/press-releases/2020-05-21-gartner-says-50-percent-of-us-healthcare-providers-will-invest-in-rpa-in-the-next-three-years
[19] https://www.idc.com/getdoc.jsp?containerId=prUS46794720 
[20] https://www.mckinsey.com/industries/healthcare-systems-and-services/our-insights/the-great-acceleration-in-healthcare-six-trends-to-heed 
[21] https://www.mckinsey.com/industries/healthcare-systems-and-services/our-insights/the-next-wave-of-healthcare-innovation-the-evolution-of-ecosystems 
[22] https://www.cbinsights.com/research/internet-of-medical-things-5g-edge-computing-changing-healthcare/
[23] https://siliconangle.com/2020/12/08/future-state-edge-computing/
[24] https://www.idc.com/getdoc.jsp?containerId=prUS46878020
[25] https://www.mckinsey.com/industries/healthcare-systems-and-services/our-insights/telehealth-a-quarter-trillion-dollar-post-covid-19-reality
[26] https://go.forrester.com/blogs/will-virtual-care-stand-the-test-of-time-if-youre-asking-the-question-its-time-to-catch-up/
[27] https://www.computerworld.com/article/3529427/how-iot-is-becoming-the-pulse-of-healthcare.html

 

What are Time Series Database Use Cases?

What are Time Series Database Use Cases?

What do self-driving cars, smart homes, autonomous stock/crypto trading algorithms, or energy sensor systems have in common? These applications are all based on a form of data that measures how things change over time. It’s called time-series data and it plays a very important role in our lives today.

Accordingly, time-series databases also became a hot topic.

time series database use cases

What is a time-series database?

A time-series database (TSDB) can be defined simply as a database optimized for storing and using time-stamped or time-series data. You don’t need to use a TSDB to work with time-series data. Any relational or NoSQL database or a key-value-store will do, e.g. MongoDB or redis. However, when dealing with time-series data (e.g. temperature, air pressure or car velocity data), a TSDB makes your life as a developer a hell of a lot easier.

Indeed, the two main reasons why TSDBs is the fastest-growing category of databases, are usability and scalability. A purpose-built time-series database typically includes common functions of time-series data analysis, which is convenient when working with time-series data. Because time-series data typically continually produces new data entries, data grows pretty quickly, and with high-frequency data or many time-series data sources, data ingestion quickly becomes a challenge. Time-series databases are optimized to scale well for time-series data with time being a common denominator and outperform any other database without specific time-series optimizations. This is why more and more people are adopting time-series databases and using them for a variety of use cases.

What are time-series database use cases?

Monitoring Use Case time series

Monitoring sensor data 

One of the use cases is the monitoring of sensor data for safety measurements, predictive maintenance, or assistance functions. E.g. a car stores and uses all kinds of sensor data like tyre pressure, surrounding temperature and humidity for driver assistance and maintenance support. An aircraft monitors gravity and aerodynamic principles to reassure pilots that everything is alright – or to alert them that something has gone wrong. In fact, a Boeing creates on average half a terabyte of data per flight, most of which is time-series data.  [1]

Logistics Use Case time series database

Tracking assets

Tracking assets is ideal for a time-series database as you constantly want to monitor where assets are, e.g. the cars of a fleet or any goods you might be stocking or shipping. These applications typically include unique vehicle or asset IDs, GPS coordinates, and additional metadata per timestamp. Apart from keeping track of the assets in realtime, you also can use the data for logistics and optimize e.g. your stocking and delivery processes.

edge time series ecommerce

Analyzing and predicting shopping behavior

Or, many e-commerce systems store all information of an item from product inventory, logistics data and any available environmental data to transaction amount, all items of the shopping cart purchased, to payment data, order information etc. In this case, a TSDB will be used to collect these large amounts of data and analyze them quickly to determine e.g. what to recommend to customers to buy next or optimize the inventory or predict future shopping behavior.

What are the most popular time series databases?

Well, here is our list of popular / established time series databases to use in 2020 to get you started:

  • InfluxDB: an open-source time series database, written in Go and optimized for high-availability storage and retrieval of time series data for operations monitoring, application metrics, IoT sensor data, and real-time analytics
  • KairosDB: a fast distributed scalable time series database written on top of Cassandra. 
  • Kdb+:  is a column-based relational time series database with a focus on applications in the financial sector.
  • Objectbox TS: superfast object persistence with time-series data on the edge. Collect, store, and query time-series data on the edge and sync selective data to / from a central location on-premise or in the cloud as needed.
  • TimescaleDB: an open-source database designed to make SQL scalable for time-series data. It is engineered up from PostgreSQL and packaged as a PostgreSQL extension with full SQL support.

For an overview of time-series databases currently available for productive use, see DB Engines. The database of databases is also a good resource if you are deeply interested in the database landscape; it is more extensive, but it includes any DB available independent of the level of support or if it is still maintained, also hobby projects. 

Time Series Database Use Cases

What do you do when you have more than just time-series data?

Typically, a time-series database is not well suited to model non-time-based data. Therefore, many companies choose to implement two databases. This increases overhead, disk space, and is especially impractical when you deal with edge devices. 

Time Series + Object-Oriented Data Persistence

Storing and processing both time series data and objects, developers can collect complex datasets and combine them with time-series data. Combining these data types gives a more complete understanding and context to the data – not just what happens over time, but also other factors that affect the results. 

The best option is a robust object-oriented database solution that lets you model your data as it reflects the factual use case / the real world in objects and on-top is optimized for time series data. You can model your world in objects and combine this with the power of time-series data to identify patterns in your data. If this is indeed a database optimized for restricted devices and Edge Computing, you can even use this data in real-time and on the device. By combining time series data with more complex data types, an object time-series edge database can empower new use cases on the edge based on a fast and easy all-in-one data persistence solution. 

Still have questions? Feel free to contact us here!

—–

[1] Time Series Management Systems: A Survey Søren Kejser Jensen, Torben Bach Pedersen, Senior Member, IEEE, Christian Thomsen

What Drives Edge Computing?

What Drives Edge Computing?

Data is exploding in every respect: in data volume, data velocity, and data variety (the 3 v’s). One driver of this phenomenon is the growing number of Mobile and IoT devices and thus, data sources. Making this data useful is one of the driving forces behind the adoption of Edge Computing. New use cases don’t only rely on using this data, but also upon the usability and speed of usability of this ever growing data. There are several practical challenges with this growing data volume that drive the adoption of Edge Computing:

New Use Cases Drive Edge Computing

what-drives-edge-computing

Bandwidth Limitations

The existing network infrastructure cannot support sending all the data to the cloud. Particularly in urban areas there is a concentration of devices and data overburdens existing infrastructure. While 5G promises some relief, it is no hailbringer. First of all, if you want to implement your IoT project now, 5G is not yet available and many questions about 5G remain, e.g. pricing. But moreover, as the number of devices and data is growing ever faster, it is already clear that data volumes will outpace what 5G can support. Edge Computing will be an important technology alongside 5G to enable IoT.

Fast Data Requirements  

Response time requirements are growing at the same time as data volumes are increasing. Sending data to the cloud for computation and storage means applications’ response times have a higher latency and depend on the network, which cannot guarantee response rates. Edge computing guarentees significantly faster data. Use cases that need speedy response times or guaranteed responses cannot rely on cloud computing. For example, in driver assistance, where every millisecond counts or in factory floors, where downtimes are too costly.

Sustainability

Sending data to the cloud and storing it there is inefficient and therefore costly – not only in plain €, but with regards to CO2 emissions too. The distance the data needs to travel needs hardware, connectivity and electric power. Therefore, sending data unnecessarily back and forth is wasteful beaviour and burdens the environment unnecessarily. With growing data volumes, that burden is growing. In fact, analysts predict  that cloud computing data centers will consume as much as 21% of the total global energy by 2030. [1]

To scale your prototype, you need to move to the edge

At the start of IoT projects, quick prototyping, testing and piloting on early iterations of an application’s functionalities, can effectively be done in the cloud. However, in productive environments when applications scale it is often hard or impossible to keep cloud costs at scale, making the business not viable. Then it is time to move to the edge.

At the same time, decreasing hardware costs and hardware sizes are enabling more complex local computing, meaning there is less need for additional cloud usage. E.g. increasingly AI and ML is done at the edge, including model training.

data-volume-edge-computing-solution-iot-mobile

Data accessibility and Smart Syncing

Today’s successful businesses require a smarter approach to data management and integration. Data synchronization increases operational efficiencies, saving time and resources by eliminating redundant data transfer. With data synchronization, only predefined, useful parts of a data set are sent to a central instance. This means that while large volumes of data can be collected and analyzed locally, not all of this data is sent to and saved in the cloud. This reduces the impact on bandwidth, utilizes the local hardware resources for fast guaranteed response times, and keeps project cloud costs low – ultimately creating a more sustainable and efficient model of data architecture, enabling long term project scalability. 

ObjectBox’ current database technology is enabling companies to persist and use data on edge devices, faster than any alternative on the market. It enables networks of edge devices working without a central instance, enabling even more new use cases.

How Building Green IoT Solutions on the Edge Can Help Save Energy and CO2

How Building Green IoT Solutions on the Edge Can Help Save Energy and CO2

The internet of things (IoT) has a huge potential to reduce carbon emissions, as it enables new ways of operating, living, and working [1] that are more efficient and sustainable. However, IoT’s huge and growing electricity demands are a challenge. This demand is due primarily to the transmission and storage of data in cloud data centers. [2] While data center efficiency and the use of green energy will reduce the CO2 emissions needed for this practice, it is not addressing the problem directly. [3

iot-data-cloud-energy-waste

With ObjectBox, we address this unseen and fast-growing CO2 source at the root: ObjectBox empowers edge computing, reducing the volume of data transmitted to central data storage, while at the same time, heightening data transmission and storage efficiency. [4] We’ve talked before about how edge computing is necessary for a sustainable future, below we dive into the numbers a bit deeper. TLRD: ObjectBox enables companies to cut the power consumption of their IoT applications, and thus their emissions, by 50 – 90%. For 2025, the potential impact of ObjectBox is a carbon emission reduction of 594 million metric tons (see calculations below).

How ObjectBox’ Technology Reduces Overall Data Transmission

 ObjectBox reduces data transmission in two ways: 1. ObjectBox reduces the need for data transmission, 2. ObjectBox makes data transmission more efficient. ObjectBox’ database solution allows companies to build products that store and process data on edge devices and work with that data offline (as well as online). This

Green IoT Solution

not only improves performance and customer experience, it also reduces the overall volume of data that is being sent to the cloud, and thus the energy needed to transfer the data as well as store it in the cloud. ObjectBox’ Synchronization solution makes it easy for companies to transmit only the data that needs to be transmitted through 1) selective two-way syncing and 2) differential delta syncing. Synchronizing select data reduces the energy required for unnecessarily transmitting all data to the cloud.

We have demonstrated in exemplary case studies that ObjectBox can reduce total data transmissions by 70-90%, depending on the case. There will, however, typically be value in transmitting some parts of data to a central data center (cloud); ObjectBox Sync combines efficient compression based on standard and proprietary edge compression methods to keep this data small. ObjectBox also has very little overhead. Comparing the transmission of the same data sets, ObjectBox saves 40-60% on transmission data volume through the delta syncing and compression, and thus saves equivalent CO2 emissions for data transmissions. Additional studies support these results, and have shown that moving from a centralized to a distributed data structure, saves between 32 and 93% of transmission data. [5

sync-sustainable-data-save-energy

Calculations: How Does ObjectBox Save CO2?

Physically using a device consumes little energy directly; it is the wireless cloud infrastructure in the backend (data center storage and data transmission) that is responsible for the high carbon footprint of mobile phones [6] and IoT devices. Estimates say that IoT devices will produce around 2,8 ZB of data in 2020 (or 2,823,000,000,000  GB), globally. [7] Only a small portion of that data actually gets stored and used; we chose to use a conservative estimate of 5% [8] (141,150,000,000 GB) and of that portion, 90% is transferred to the cloud [9] (127,035,000,000 GB). Transferring 1 GB of data to the cloud and storing it there costs between 3 and 7 kWh. [10] Assuming an average of 5 kWh this means a 127,035,000,000 GB multiplied by 5kWh, resulting in a total energy expenditure of 635,175,000,000 kWh. Depending on the energy generation used, CO2 emissions vary. We are using a global average of 0,475 kgCO2 / 1 kwH. [11] In total this means that there will be 301,708,125,000 KG of CO2, or roughly 301 million metric tons of CO2 produced to transfer data to the cloud and store it there in 2020. 

Projections for 2025 have data volumes as high as 79.4 ZB. [12] Following the same calculations as above, IoT devices would be responsible for 8 billion metric tons of CO2 in 2025.* We estimate that using ObjectBox can cut CO2 caused by data transmission and data centers by 50-90%, by keeping the majority of data on the device, and transmitting data efficiently. It will take time for ObjectBox to enter the market, so assuming a 10% market saturation by 2025 and an average energy reduction of 70%, using ObjectBox could cut projected CO2 emissions by 594 million metric tons in 2025.

ObjectBox is on a mission to reduce digital waste which unnecessarily burdens bandwidth infrastructure and fills cloud servers, forcing the expansion of cloud farms and in turn, contributing to the pollution of the environment. As our digital world grows, we all need to give some thought to how we should structure our digital environments to optimize and support useful, beneficial solutions, while also keeping them efficient and sustainable. 

*Of course, in that time, the technologies will all be more efficient and thus use less electricity while at the same time CO2 emissions / kWh will have dropped too. Thus, we are aware that this projection is an oversimplification of a highly complex and constantly changing system.

[1] https://www.theclimategroup.org/sites/default/files/archive/files/Smart2020Report.pdf
[2] https://www.iea.org/reports/tracking-buildings/data-centres-and-data-transmission-networks
[3]“Data centres… have eaten into any progress we made to achieving Ireland’s 40% carbon emissions reduction target.” from https://www.climatechangenews.com/2017/12/11/tsunami-data-consume-one-fifth-global-electricity-2025/
[4] https://medium.com/stanford-magazine/carbon-and-the-cloud-d6f481b79dfe
[5] https://www.researchgate.net/publication/323867714_The_carbon_footprint_of_distributed_cloud_storage
[6] https://www.resilience.org/stories/2020-01-07/the-invisible-and-growing-ecological-footprint-of-digital-technology/
[7] https://www.idc.com/getdoc.jsp?containerId=prUS45213219, https://priceonomics.com/the-iot-data-explosion-how-big-is-the-iot-data/, https://www.gartner.com/en/newsroom/press-releases/2018-11-07-gartner-identifies-top-10-strategic-iot-technologies-and-trends, https://www.iotjournaal.nl/wp-content/uploads/2017/02/white-paper-c11-738085.pdf, ObjectBox research
[8] Forrester (https://internetofthingsagenda.techtarget.com/blog/IoT-Agenda/Preventing-IoT-data-waste-with-the-intelligent-edge), Harvard BR (https://hbr.org/2017/05/whats-your-data-strategy), IBM (http://www.redbooks.ibm.com/redbooks/pdfs/sg248435.pdf), McKinsey (https://www.mckinsey.com/business-functions/mckinsey-digital/our-insights/the-internet-of-things-the-value-of-digitizing-the-physical-world)
[9] https://www.gartner.com/smarterwithgartner/what-edge-computing-means-for-infrastructure-and-operations-leaders/
[10] According to the American Council for an Energy-Efficient Economy: 5,12 kWh of electricity / GB of transferred data. According to a Carnegie Mellon University study: 7 kWh / GB. The American Council for an Energy-Efficient Economy concluded: 3.1 kWh / GB.
[11] https://www.iea.org/reports/global-energy-co2-status-report-2019/emissions
[12] https://www.idc.com/getdoc.jsp?containerId=prUS45213219