IoT, Edge Computing, and Digitalization in Healthcare

IoT, Edge Computing, and Digitalization in Healthcare

The healthcare industry is experiencing an unprecedented surge in data generation, responsible for approximately 30% of the world’s total data volume. This vast and fast-growing amount of health data is the primary force behind the digital transformation of healthcare. Only through the adoption of advanced technologies can healthcare providers manage, analyze, and secure this information. While COVID-19 accelerated this shift, contributing to the explosion of health data, the ongoing demand for real-time patient insights, personalized treatment, and improved operational efficiency continues to drive the sector toward digitalization and AI. Simultaneously, growing data privacy concerns, increasing costs, and heavier regulatory requirements are challenging the use of cloud computing to manage this data. A megashift to Edge Computing and Edge AI is addressing these challenges, enabling a faster, safer, and more reliable digital healthcare infrastructure.

The digital healthcare market 2024 and beyond, a high-speed revolution

Prior to COVID, growth in digital health adoption stalled. However, digitalization in the healthcare industry has sky-rocketed since the start of the pandemic. Reflecting this market turnaround, followed by the rise of advanced digital tools like AI, recent years have been record-breaking for investments in healthcare companies. A trend that will continue in the next years, as analysts predict rapid growth across digital healthcare market sectors:

Healthcare market overview

Drivers of growth and change in digital healthcare

 

Digital Healthcare Growth Driver 1: Growing Medical IoT Device Adoption

There will be a projected 40 billion IoT devices by 2030. IoMT devices already accounted for 30% of the entire IoT device market in 2020. Internet of Medical Things (IoMT) are hardware devices designed to process, collect, and/or transmit healthrelated data via a network. According to Gartner, 79% of healthcare providers are already using IoT in their processes, i.e. remote health monitoring via wearables, ingestible sensors, disinfection robots, or closed-loop insulin delivery systems. IoMT devices increase safety and efficiency in healthcare, and future technical applications, like smart ambulances or augmented reality glasses that assist during surgery, are limitless.

IoMT devices accounted for 30% of the IoT device market

health-care-edge-computing

Digital Healthcare Growth Driver 2: The Explosion of Health Data

Growing IoMT adoption is subsequently driving a rapid increase in the amount of collected health data. According to the RBC study, the healthcare industry is now responsible for approximately 30% of the world’s total data volume. By 2025, healthcare data is expected to continue growing at a 36% CAGR, outpacing data volumes from sectors like manufacturing, financial services, and media. Big health data sets are being used to revolutionize healthcare, bringing new insights into fields like oncology, and improving patient experience, care, and diagnosis. According to the Journal of Big Data: “taken together, big data will facilitate healthcare by introducing prediction of epidemics (in relation to population health), providing early warnings of disease conditions, and helping in the discovery of novel biomarkers and intelligent therapeutic intervention strategies for an improved quality of life.” In fact, the healthcare analytics market is projected to reach $129.7 billion by 2028, growing at a 23.5% CAGR. This growth is driven by the need for real-time data processing, personalized medicine, and predictive analytics to manage chronic conditions and optimize hospital operations.

health-care-edge-computing

Healthcare data occupies ~30% of the world’s total data volume

Digital Healthcare Growth Driver 3: Artificial Intelligence

The increase in healthcare data opens up new opportunities and challenges to apply advanced technologies like big data analytics and artificial intelligence (AI) to improve healthcare delivery, patient outcomes, and operational efficiency. For instance, AI is being used to analyze medical imaging data, identify patterns in electronic health records, and predict patient outcomes, contributing to improved patient care. By 2026, AI is projected to save the global healthcare industry over $150 billion annually, by answering “20 percent of unmet clinical demand.” 

Generative AI, which includes Large Language Models (LLMs) such as GPT-4, is playing a crucial role in this transformation. According to the survey from McKinsey, 70% of surveyed healthcare organizations are either currently testing or actively using generative AI tools for both clinical and administrative applications. This is unsurprising, as LLM Chatbots can reduce waiting times by 80% in healthcare facilities. In diagnostics, LLMs are being applied to interpret electronic health records and assist with predictive analytics, leading to a reduction in hospital readmissions by up to 22%. Additionally, LLMs have helped improve medication adherence rates by 60%, demonstrating their impact on patient care quality.

70% of healthcare organizations plan or use AI

health-care-edge-computing

Digital Healthcare Growth Driver 4: Artificial Intelligence

With the rise of IoMT and the boost in healthcare data, Edge Computing is becoming a key driver of healthcare digitalization. The majority of IoMT devices (55.3 %) currently operate on-premise rather than in the cloud, ensuring faster, more secure real-time data processing. This shift to Edge Computing enhances data privacy and reduces latency, which is critical in life-critical medical applications. Additionally, the development of Small Language Models (SLMs) for on-device AI (Edge AI) allows healthcare providers to deploy AI-powered solutions directly on medical devices. This helps with tasks like remote monitoring and diagnostics without the need for cloud connectivity, which is particularly beneficial in environments with limited internet access

As IoMT continues to evolve, Edge Computing will play an essential role in supporting healthcare’s increasing demand for real-time data processing. By 2025, it is projected that 75% of the healthcare data will be generated at the Edge, further driving the adoption of these technologies across the industry.

AI-Health-Icon

75% of the healthcare  data will be generated at the Edge in 2025

Digital Healthcare Growth Driver 5: Underlying Social Megatrends

The global population is growing; global life expectancy is rising. Accordingly, by 2030 the world needs more energy, more food, and more water. Explosive population growth in some areas versus declines in others contributes to shifts in economic power, resource allocation, societal habits, and norms. Many Western populations are aging rapidly. E.g. in America, the number of people 65+ is expected to nearly double to 72.1 million by 2034. Because the population is shrinking at the same time, elder care is a growing challenge and researchers are looking to robots to solve it

Health megatrends focus not only on the prevention of disease, but also on the perception of wellness, and new forms of living and working. Over this decade more resources will be spent on health and longevity, leading to artificially and technologically enhanced human capabilities. More lifestyle-related disorders and diseases are expected to emerge in the future.

A focus on health and longevity will
lead to artificial & tech-enhanced
human capabilities

health-care-edge-computing

The Challenges of Healthtech

Along with more data, more devices, and more opportunities also comes more responsibility and more costs for healthcare providers.

health-care-edge-computing

Data Volume and Availability With the growing number of digital healthcare and medical devices, a dazzling volume of health data is created and collected across many different channels. It will be vital for the healthcare industry to reliably synchronize and combine data across devices and channels. Due to the sheer volume, reliable collection and analysis of this data is a major challenge. After it’s been processed, data needs to be available on demand, i.e. in emergency situations that require reliable, fast, available data.

health-care-edge-computing

Reliability, Privacy, and Data Security are extremely important in health technology; 70% of healthcare consumers are concerned about data privacy. Data use is often governed by increasingly strict national regulations, i.e. HIPAA (USA) and/or GDPR (Europe). With the number of cyber-attacks in the healthcare industry on the rise, healthcare professionals must be even more diligent about the storage and processing of data. In addition, healthtech must be extremely well vetted; failures can cost lives – typical “banana products”, which ripen with the customers, are a no-go.

health-care-edge-computing

IT Costs Medical devices contribute a large portion to healthcare budgets. However as data volumes grow, data costs will also become a relevant cost point. Sending all health data to the cloud to be stored and processed is not only slow and insecure, it is also extremely costly. To curb mobile network and cloud costs, much health data can be stored and processed at the edge, on local devices, with only necessary data being synced to a cloud or central server. By building resilient data architecture now, healthcare providers (e.g. hospitals, clinics, research centers) can avoid future costs and headaches.

Edge Computing is Integral to Data-driven Healthcare Ecosystems

With big data volumes, industries like healthcare need to seek out resilient information architectures to accommodate growing numbers of data and devices. To build resilient and secure digital infrastructure, healthcare providers will need to utilize both cloud computing and edge computing models, exploiting the strengths of both systems.

Cloud & Edge: What’s the Difference?

Cloud Computing information is sent to a centralized data center, to be stored, processed and sent back to the edge. This causes latency and a higher risk of data breaches. Centralized data is useful for large-scale data analysis and the distribution of data between i.e. hospitals and doctors’ offices.

Edge Computing Data is stored and processed on or near the device it was created on. Edge Computing works without an internet connection, and thus is reliable and robust in any scenario. It is ideal for time-sensitive data (real-time), and improved data privacy and security.

health-care-edge-computing

Edge Computing contributes to resilient and secure healthcare data systems

health-care-edge-computing

Transforming Healthcare with Edge Computing

Use Case: Secure and Up to Date Digital Record Keeping in Doctors Offices

For private doctors’ offices, embracing digitalization comes with different hurdles than larger healthcare providers. Often, offices do not keep a dedicated IT professional on staff, and must find digital solutions that serve their needs, while allowing them to comply with ever-increasing data regulations. As an industry used to legislative challenges, GPs know that sensitive patient data must be handled with care.

Solution providers serving private doctors’ offices are using edge databases to help keep patient data secure. An edge database allows private GPs to collect and store digital data locally. In newer practice setups, doctors use tablets, like iPads, throughout their practice to collect and track patient data, take notes and improve flexibility. This patient data should not be sent or stored in a central cloud server as this increases the risk of data breaches and opens up regulatory challenges. In a cloud-centered setup, the doctor also always needs to rely on a constant internet connection being available, making this also a matter of data availability

health-care-edge-computing

Accordingly, the patient data is stored locally, on the iPads, accessible only by the doctor treating the patient. Some of the data is synchronized to a local, in-office computer at the front desk for billing and administration. Other data is only synchronized for backup purposes and encrypted. Such a setup also allows synchronizing data between iPads, enabling doctors to share data in an instant.

Use Case: Connected Ambulances – Real-Time Edge Data from Home to Hospital

health-care-edge-computing

Between an incidence location and the hospital, a lot can happen. What if everything that happened in the ambulance was reliably and securely tracked and shared with the hospital, seamlessly? There are already trials using 5G technology to stream real-time data to hospitals, allowing ambulance medics to access patient data while in transit. Looking to the future, Edge Computing will enable digital healthcare applications to function in real-time and reliably anywhere and anytime, e.g. a moving ambulance, in the tunnel, or a remote area, enabling ambulance teams and doctors to give the best treatment instantly / on-site, while using available bandwidth and networks when available to seamlessly synchronize the relevant information to the relevant healthcare units, e.g. the next hospital. This will decrease friction, enhance operational processes, and improve time to treatment.

Digital Healthcare: Key Take-Aways

Digital healthcare is a fast-growing industry; more data and devices alongside new tech are empowering rapid advances. Finding ways to utilize growing healthcare data, while ensuring data privacy, security and availability are key challenges ahead for healthcare providers. The healthcare industry must find the right mix of technologies to manage this data, utilizing cloud for global data exchange and big data analytics, while embracing Edge Computing for it’s speed, security, and resilience.

health-care-edge-computing

Underutilized data plays a major role in health-tech innovation, data is the lifeline of future healthcare offerings; however, there is still much work to be done to improve the collection, management, and analysis of this data.

health-care-edge-computing

It’s all about data availability. Either in emergency situations, or simply to provide a smooth patient experience, data needs to be fast, reliable, and available: when you need it where you need it.

health-care-edge-computing

Edge computing alongside other developing technologies like 5G or Artificial Intelligence will empower a new and powerful digital healthcare ecosystem.

ObjectBox provides edge data software, to empower scalable and resilient digital innovation on the edge in healthcare, automotive, and manufacturing. ObjectBox’ edge database and data synchronization solution is 10x faster than any alternative, and empowers applications
that respond in real-time (low-latency), work offline without a connection to the cloud, reduce energy needs, keep data secure, and lower mobile network and cloud costs.

The Critical Role of Databases for Edge AI

The Critical Role of Databases for Edge AI

Edge AI vs. Cloud AI

Edge AI is where Edge Computing meets AI

What is Edge AI? Edge AI (also: “on-device AI”, “local AI”) brings artificial intelligence to applications at the network’s edge, such as mobile devices, IoT, and other embedded systems like, e.g., interactive kiosks. Edge AI combines AI with Edge Computing, a decentralized paradigm designed to bring computing as close as possible to where data is generated and utilized.

What is Cloud AI? As opposed to this, cloud AI refers to an architecture where applications rely on data and AI models hosted on distant cloud infrastructure. The cloud offers extensive storage and processing power.

An Edge for Edge AI: The Cloud 

 

Cloud AI to Edge AI architecture

Example: Edge-Cloud AI setup with a secure, two-way Data Sync architecture

Today, there is a broad spectrum of application architectures combining Edge Computing and Cloud Computing, and the same applies to AI. For example, “Apple Intelligence” performs many AI tasks directly on the phone (on-device AI) while sending more complex requests to a private, secure cloud. This approach combines the best of both worlds – with the cloud giving an edge to the local AI rather than the other way around. Let’s have a look at the advantages on-device AI brings to the table.

Benefits of Local AI on the Edge

  • Enhanced Privacy. Local data processing reduces the risk of breaches.
  • Faster Response Rates. Processing data locally cuts down travel time for data, speeding up responses.
  • Increased Availability. On-device processing makes apps fully offline-capable. Operations can continue smoothly during internet or data center disruptions.
  • Sustainability/costs. Keeping data where it is produced and used minimizes data transfers, cutting networking costs and reducing energy consumption—and with it, CO2 emissions.

Challenges of Local AI on the Edge

  • Data Storage and Processing: Local AI requires an on-device database that runs on a wide variety of edge devices (Mobile,IoT, Embedded) and performs complex tasks such as vector search locally on the device with minimal resource consumption.
  • Data Sync: It’s vital to keep data consistent across devices, necessitating robust bi-directional Data Sync solutions. Implementing such a solution oneself requires specialized tech talent, is non-trivial and time-consuming, and will be an ongoing maintenance factor. 
  • Small Language Models: Small Language Models (SLMs) like Phi-2 (Microsoft Research), TinyStories (HuggingFace), and Mini-Giants (arXiv) are efficient and resource-friendly but often need enhancement with local vector databases for better response accuracy. An on-device vector database allows on-device semantic search with private, contextual information, reducing latency while enabling faster and more relevant outputs. For complex queries requiring larger models, a database that works both on-device and in the cloud (or a large on-premise server) is perfect for scalability and flexibility in on-device AI applications.

On-device AI Use Cases

On-device AI is revolutionizing numerous sectors by enabling real-time data processing wherever and whenever it’s needed. It enhances security systems, improves customer experiences in retail, supports predictive maintenance in industrial environments, and facilitates immediate medical diagnostics. On-device AI is essential for personalizing in-car experiences, delivering reliable remote medical care, and powering personal AI assistants on mobile devices—always keeping user privacy intact.

Personalized In-Car Experience: Features like climate control, lighting, and entertainment can be adjusted dynamically in vehicles based on real-time inputs and user habits, improving comfort and satisfaction. Recent studies, such as one by MHP, emphasize the increasing consumer demand for these AI-enabled features. This demand is driven by a desire for smarter, more responsive vehicle technology.

Remote Care: In healthcare, on-device AI enables on-device data processing that’s crucial for swift diagnostics and treatment. This secure, offline-capable technology aligns with health regulations like HIPAA and boosts emergency response speeds and patient care quality.

Personal AI Assistants: Today’s personal AI assistants often depend on the cloud, raising privacy issues. However, some companies, including Apple, are shifting towards on-device processing for basic tasks and secure, anonymized cloud processing for more complex functions, enhancing user privacy.

ObjectBox for On-Device AI – an edge for everyone

Edge Cloud spectrum

The continuum from Edge to Cloud

ObjectBox supports AI applications from Edge to cloud. It stands out as the first on-device vector database, enabling powerful Edge AI on mobile, IoT, and other embedded devices with minimal hardware needs. It works offline and supports efficient, private AI applications with a seamless bi-directional Data Sync solution, completely on-premise, and optional integration with MongoDB for enhanced backend features and cloud AI.

 Interested in extending your AI to the edge? Let’s connect to explore how we can transform your applications.

Bi-directional Offline-First Data Sync with MongoDB and ObjectBox

Bi-directional Offline-First Data Sync with MongoDB and ObjectBox

In today’s fast-paced, decentralized world valuable data is generated by everything, everywhere, and all at once. To harness the vast opportunities offered by this data for data-driven organizations and AI applications, you need to be able to access the data and seamlessly distribute it to when and where it’s needed.

The key to achieving this lies in efficient, offline-first on-device data storage, reliable bi-directional data sync, and a scalable data management backend in the cloud. In other words, you need the infrastructure to manage data flows bi-directionally to tap into fresh data throughout your organization, processes, and applications at the right time.

Together, MongoDB and ObjectBox provide developers with a robust solution to empower seamless workload and data flows on the edge and from the edge to the cloud. ObjectBox seamlessly syncs data bi-directionally across devices even without Internet and syncs back to the cloud and MongoDB when connected. With ObjectBox devices stay in sync also in environments with intermittent connectivity, high latency, or flaky networks. Capture and unlock the value of all your data, anytime, anywhere, without relying on a constant Internet connection, with MongoDB + ObjectBox.

Seamless Offline-First Data Sync for Edge Devices

Maintaining service continuity is essential, even when devices are offline. Your customers, users, operations, and employees need to be able to rely on essential data at all times. That’s where ObjectBox comes in. It comprises of two key components: the ObjectBox Database and ObjectBox Data Sync.

The ObjectBox Database is a lightweight, on-device solution that is highly resource-efficient and fast on restricted hardware like mobile, IoT, and embedded devices, and even in the cloud. 

ObjectBox Data Sync enables seamless bi-directional data synchronization between devices. By handling only incremental changes in a compressed binary format, ObjectBox Sync ensures minimal data transfer, automatic conflict resolution, and a seamless user experience even in fluctuating network conditions. This approach effectively simplifies the development process by offering complex sync logic via easy native-language APIs, allowing developers to focus on core app functionality.

Once a connection is available, ObjectBox Data Sync instantly synchronizes changes with MongoDB, providing real-time, bi-directional data sync between edge devices and MongoDB’s robust cloud backend.

The Benefits of Offline-First and Real-Time Data Sync with MongoDB and ObjectBox:

  • Resource-efficiency & Highspeed: ObjectBox excels at consuming minimal computational resources (CPU, power, memory, …) while delivering data persistence speed that is typically on-par with in-memory caches for read operations.
  • Offline-First Operation: Ensure continuous app performance, even with no internet connection. ObjectBox stores and syncs data bi-directionally on the edge and additionally with MongoDB once connected.
  • Real-Time Data Sync: Get reliable, bi-directional data synchronization across devices and MongoDB, enabling real-time updates and data consistency.
  • Scalable Edge: Easily handle 100k operations / second on a single device. Host the Sync server on any edge device (like a phone) and easily handle 3M clients with a three-node cluster.  
  • Scalable Cloud Backend: With MongoDB, businesses can scale their applications to handle growing data and performance demands, seamlessly syncing data between millions of devices and the cloud.

Flexible Setup Scenarios: Tailor Data Sync to Your Needs

ObjectBox and MongoDB offer flexible setup scenarios to meet different application needs. The two main setup options are the central sync and the edge sync setup.

The Central Sync Setup syncs data between edge devices and MongoDB in the cloud, providing centralized data management while retaining offline-first functionality. The ObjectBox Sync Server runs in the cloud or on-premise.

The Edge Sync Setup allows devices to operate and sync data efficiently offline between ObjectBox instances within an edge, e.g. within one location, or within a car. When reconnected, changes are synchronized back to MongoDB making it ideal for environments with intermittent connectivity or distributed devices that need to function independently while syncing back to the cloud when possible.

This structure offers a flexible approach to integrating edge and cloud systems, empowering organizations to choose the setup that best fits their specific use case. More details.

Use Cases for MongoDB + ObjectBox :

  1. Data-Driven Organizations: In a data-driven organization, every decision relies on access to relevant, up-to-date data. ObjectBox enables real-time data collection and synchronization from edge devices, ensuring access to critical data, even when devices are intermittently connected. This streamlines operations, improves decision-making, and enhances analysis across distributed teams and IoT systems. With MongoDB’s scalable cloud infrastructure, decentralized data integrates seamlessly with the cloud backend for efficient management.
  2. Point-of-Sale (PoS) & Retail Edge Computing: A seamless customer experience and the ability to keep selling and never lose a transaction, even during internet outages, are essential for PoS systems / in retail. ObjectBox enables offline-first data storage and syncing for PoS systems, allowing transactions to be processed locally, even without internet connectivity. When connectivity returns, ObjectBox syncs transaction data back to MongoDB in real time, ensuring data consistency across multiple locations. Retailers can then leverage MongoDB’s analytics to gain insights into customer behavior and optimize inventory management.
  3. Software-Defined Vehicle (SDV) & Connected Cars: Modern vehicles generate vast amounts of data from sensors and onboard systems. ObjectBox enables efficient on-device storage and processing, providing real-time access to data for navigation, diagnostics, and infotainment systems. ObjectBox Data Sync ensures that local data is synced back to MongoDB when connectivity is available, supporting centralized analytics, fleet management, and predictive maintenance, optimizing performance and safety while enhancing the user experience.
  4. Manufacturing & Smart Shopfloor Apps: In smart factories, machines and sensors continuously generate data that must be analyzed and processed in real time. ObjectBox enables local data storage and fast data sync on-premise without the necessity for an Internet connection, ensuring that critical systems that are not connected to the Internet can run smoothly on-site. With a connected instance, ObjectBox takes care of synchronizing this data with the cloud and MongoDB for further analysis and central dashboards.
  5. AI-Applications with On-device Vector Search: ObjectBox is the first and only on-device vector database, empowering developers to run AI locally on mobile, IoT, embedded, and other commodity devices (Edge AI). In combination with a Small Language Model (SLM), this allows developers to build local AI applications (e.g. RAG, genAI) that run directly on the device—without needing a cloud connection. By syncing with MongoDB, businesses can combine the power of on-device AI with centralized cloud data for even greater insights and performance. This is especially beneficial in scenarios requiring real-time decision-making, such as personalized customer experiences and predictive maintenance.

In today’s data-driven world, a data-first strategy requires seamless integration between edge and cloud data management. The combination of MongoDB and ObjectBox unlocks the full potential of your data. MongoDB’s powerful cloud platform, together with ObjectBox’s efficient on-device database and offline-first capabilities, is ideal for capturing the value of your data from anywhere, including distributed edge devices where valuable data is generated all the time. This partnership empowers businesses to seamlessly handle decentralized data, enabling fast and reliable operations at the edge while syncing back to the cloud for centralized management. Whether on IoT devices, mobile, embedded systems, or commodity hardware, ObjectBox and MongoDB ensure optimal performance everywhere. From remote areas to bad networks, our joint solution keeps data flowing reliably between the edge and the MongoDB backend, even when connectivity or nodes are lost.

The rise of small language models

The rise of small language models

As artificial intelligence (AI) continues to evolve, companies, researchers, and developers are increasingly recognizing that bigger isn’t always better. Therefore, the era of ever-expanding model sizes is giving way to more efficient, compact models, so-called Small Language Models (SLMs). SLMs offer several key advantages that address both the growing complexity of AI and the practical challenges of deploying large-scale models. In this article, we’ll explore why the race for larger models is slowing down and how SLMs are emerging as the sustainable solution for the future of AI. 

From Bigger to Better: The End of the Large Model Race

Up until 2023, the focus was on expanding models to unprecedented scales. But the era of creating ever-larger models appears to be coming to an end. Many newer models like Grok or Llama 3 are smaller in size yet maintain or even improve performance compared to models from just a year ago. The drive now is to reduce model size, optimize resources, and maintain power. 

The Plateau of Large Language Models (LLMs)

Why Bigger No Longer Equals Better

As models become larger, developers are realizing that the performance improvements aren’t always worth the additional computational cost. Breakthroughs in knowledge distillation and fine-tuning enable smaller models to compete with and even outperform their larger predecessors in specific tasks. For example, medium-sized models like Llama with 70B parameters and Gemma-2 with 27B parameters are among the top 30 models in the chatbot arena, outperforming even much larger models like GPT-3.5 with 175B parameters.

The Shift Towards Small Language Models (SLMs)

In parallel with the optimization of LLMs, the rise of SLMs presents a new trend (see Figure). These models require fewer computational resources, offer faster inference times, and have the potential to run directly on devices. In combination with an on-device database, this enables powerful local GenAI and on-device RAG apps on all kinds of embedded devices, like on mobile phones, Raspberry Pis, commodity laptops, IoT, and robotics.

Advantages of SLMs

Despite the growing complexity of AI systems, SLMs offer several key advantages that make them essential in today’s AI landscape: 

speed-icon

Efficiency and Speed
SLMs are significantly more efficient, needing less computational power to operate. This makes them perfect for resource-constrained environments like edge computing, mobile phones, and IoT systems. This enables quicker response times and more real-time applications. For example, studies show that small models like DistilBERT can retain over 95% of the performance of larger models in some tasks while being 60% smaller and faster to execute.

Accessibility
As SLMs are less resource-hungry (less hardware requirements, less CPU, memory, power needs), they are more accessible for companies and developers with smaller budgets. Because the model and data can be used locally, on-device / on-premise, there is no need for cloud infatstructure and they are also usable for use cases with high privacy requirements. All in all, SLMs democratize AI development and empower smaller teams and individual developers to deploy advanced models on more affordable hardware.

Cost Reduction and Sustainability
Training and deploying large models require immense computational and financial resources, and comes with high operational costs. SLMs drastically reduce the cost of training, deployment, and operation as well as the carbon footprint, making AI more financially and environmentally sustainable.

Gear

Specialization and Fine-tuning
SLMs can be fine-tuned more efficiently for specific applications. They excel in domain-specific tasks because their smaller size allows for faster and more efficient retraining. It makes them ideal for sectors like healthcare, legal document analysis, or customer service automation. For instance, using the ‘distilling step-by-step’ mechanism, a 770M parameter T5 model outperformed a 540B parameter PaLM model using 80% of the benchmark dataset, showcasing the power of specialized training techniques with a much smaller model size

Gear

On-Device AI for Privacy and Security
SLMs are becoming compact enough for deployment on edge devices like smartphones, IoT sensors, and wearable tech. This reduces the need for sensitive data to be sent to external servers, ensuring that user data stays local. With the rise of on-device vector databases, SLMs can now handle use-case-specific, personal, and private data directly on the device. This allows more advanced AI apps, like those using RAG, to interact with personal documents and perform tasks without sending data to the cloud. With a local, on-device  vector database users get personalized, secure AI experiences while keeping their data private.

 The Future: Fit-for-Purpose Models: From Tiny to Small to Large Language models

 The future of AI will likely see the rise of models that are neither massive nor minimal but fit-for-purpose. This “right-sizing” reflects a broader shift toward models that balance scale with practicality. SLMs are becoming the go-to choice for environments where specialization is key and resources are limited. Medium-sized models (20-70 billion parameters) are becoming the standard choice for balancing computational efficiency and performance on general AI tasks. At the same time, SLMs are proving their worth in areas that require low latency and high privacy.

Innovations in model compression, parameter-efficient fine-tuning, and new architecture designs are enabling these smaller models to match or even outperform their predecessors. The focus on optimization rather than expansion will continue to be the driving force behind AI development in the coming years.

 

 Conclusion: Scaling Smart is the New Paradigm

 

As the field of AI moves beyond the era of “bigger is better,” SLMs and medium-sized models are becoming more important than ever. These models represent the future of scalable and efficient AI. They serve as the workhorses of an industry that is looking to balance performance with sustainability and efficiency. The focus on smaller, more optimized models demonstrates that innovation in AI isn’t just about scaling up; it’s about scaling smart.

Local AI – what it is and why we need it

Local AI – what it is and why we need it

Artificial Intelligence (AI) has become an integral part of our daily lives in recent years. However, it has been tied to running in huge, centralized cloud data centers. This year, “local AI”, also known as “on-device AI” or “Edge AI”, is gaining momentum. Local vector databases, efficient language models (so-called Small Language Models, SLMs), and AI algorithms are becoming smaller, more efficient, and less compute-heavy. As a result, they can now run on a wide variety of devices, locally.

Figure 1. Evolution of language model’s size with time. Large language models (LLMs) are marked as celadon circles, and small language models (SLMs) as blue ones.

What is Local AI (on-device AI, Edge AI)?

Local AI refers to running AI applications directly on a device, locally, instead of relying on (distant) cloud servers. Such an on-deivce AI works in real-time on commodity hardware (e.g. old PCs), consumer devices (e.g. smartphones, wearables), and other types of embedded devices (e.g. robots and point-of-sale (POS) systems used in shops and restaurants). An interest in local Artificial Intelligence is growing (see Figure 2).

Figure 2. Interest over time according to Google Trends.

Why use Local AI: Benefits

Local AI addresses many of the concerns and challenges of current cloud-based AI applications. The main reasons for the advancement of local AI are: 

On top, local AI reduces:

  • latency, enabling real-time apps
  • data transmission and cloud costs, enabling commodity business cases

In short: By leveraging the power of Edge Computing and on-device processing, local AI can unlock new possibilities for a wide range of applications, from consumer applications to industrial automation to healthcare.

Privacy: Keeping Data Secure

In a world where data privacy concerns are increasing, local AI offers a solution. Since data is processed directly on the device, sensitive information remains local, minimizing the risk of breaches or misuse of personal data. No need for data sharing, and data ownership is clear. This is the key to using AI responsibly in industries like healthcare, where sensitive data needs to be processed and used without being sent to external servers. For example, medical data analysis or diagnostic tools can run locally on a doctor’s device and be synchronized to other on-premise, local devices (like e.g. PCs, on-premise servers, specific medical equipment) as needed. This ensures that patient data never leaves the clinic, and data processing is compliant with strict privacy regulations like GDPR or HIPAA.

Accessibility: AI for Anyone, Anytime

One of the most significant advantages of local AI is its ability to function without an internet connection. This opens up a world of opportunities for users in remote locations or those with unreliable connectivity. Imagine having access to language translation, image recognition, or predictive text tools on your phone without needing to connect to the internet. Or a point-of-sale (POS) system in a retail store that operates seamlessly, even when there’s no internet. These AI-powered systems can still analyze customer buying habits, manage inventory, or suggest product recommendations offline, ensuring businesses don’t lose operational efficiency due to connectivity issues. Local AI makes this a reality. In combination with little hardware requirements, it makes AI accessible for anyone, anytime. Therefore, local AI is an integral ingredient in making AI more inclusive and to democratize AI.

Sustainability: Energy Efficiency

Cloud-based AI requires massive server farms that consume enormous amounts of energy. Despite strong efficiency improvements, in 2022, data centers globally consumed between 240 and 340 terawatt-hours (TWh) of electricity. To put this in perspective, data centers now use more electricity than entire countries like Argentina or Egypt. This growing energy demand places considerable pressure on global energy resources and contributes to around 1% of energy-related CO2 emissions. The rise of AI has amplified these trends. AI workloads alone could drive a 160% increase in data center energy demand by 2030, with some estimates suggesting that AI could consume 500% more energy in the UK than it does today. By that time, data centers may account for up to 8% of total energy consumption in the United States. In contrast, local AI presents a more sustainable alternative, e.g. by leveraging Small Language Models, which require less power to train and run. Since computations happen directly on the device, local AI significantly reduces the need for constant data transmission and large-scale server infrastructure. This not only lowers energy use but also helps decrease the overall carbon footprint. Additionally, integrating a local vector database can further enhance efficiency by minimizing reliance on power-hungry data centers, contributing to more energy-efficient and environmentally friendly technology solutions.

When to use local AI: Use case examples

Local AI enables an infinite number of new use cases. Thanks to advancements in AI models and vector databases, AI apps can be run cost-effectively on less capable hardware, e.g. commodity PCs, without the need for an internet connection and data sharing. This opens up the opportunity for offline AI, real-time AI, and private AI applications on a wide variety of devices. From smartphones and smartwatches to industrial equipment and even cars, local AI is becoming accessible to a broad range of users. 

  • Consumer Use Cases (B2C): Everyday apps like photo editors, voice assistants, and fitness trackers can integrate AI to offer faster and more personalized services (local RAG), or integrate generative AI capabilities. 
  • Business Use Cases (B2B): Retailers, manufacturers, and service providers can use local AI for data analysis, process automation, and real-time decision-making, even in offline environments. This improves efficiency and user experience without needing constant cloud connectivity.

Conclusion

Local AI is a powerful alternative to cloud-based solutions, making AI more accessible, private, and sustainable. With Small Language Models and on-device vector databases like ObjectBox, it is now possible to bring AI onto everyday devices. From the individual user who is looking for convenient, always-available tools to large businesses seeking to improve operations and create new services without relying on the cloud – local AI is transforming how we interact with technology everywhere.

Vector search: making sense of search queries

Vector search: making sense of search queries

Today, finding the most valuable information to your search is more complicated than finding a needle in a haystack. Traditional search engines match keywords and favor SEO-optimized content, but what if there was a way for search engines to truly understand the meaning behind our queries? Enter vector search – a powerful technology that is transforming how we navigate information, not just for users, but also for applications performing background searches. In this article, we will discuss what vector search is and how it works.

What is a vector search and why should you care?

Example Results with a traditional search for “Simple Fruit Cake”.

Vector search, which is also known as semantic search, is a technology that improves search accuracy by understanding the meaning (semantics) of the data and relations between its parts. Unlike traditional search, vector search efficiently handles synonyms, typos, ambiguous language, and broad or fuzzy queries. This is because it focuses on meaning, not just keywords.

Imagine that you are searching for a dessert to cook during the weekend. In a traditional search engine, the “simple fruit cake” query will reveal only websites that include these keywords. However, a vector search engine is able to provide results like “apple pie in 20 minutes” or “easy summer desserts”, which capture the essence of the query and align with your desire for a straightforward dessert option, providing more valuable results to you. 

At its core, vector search uses Large Language models (LLMs), like GPT, to transform data into mathematical vectors, also known as vector embeddings

What is a vector embedding?

2D Vector Space Representation. “Easy apple pie” is close to “simple fruit cake” as they are both simple and have fruit as an ingredient. “Easy chocolate mousse” shares simplicity but does not contain fruit. “Fancy plum cake” has fruit but is not simple to make. And “extravagant chocolate mousse” does not share either simplicity or fruit as an ingredient. Thus, it is the farthest from “simple fruit cake”.

A vector or vector embedding is a numerical representation of any kind of unstructured data (e.g. texts, images, videos, audio). It captures its meaning while being easy and efficient to compute with. Think of it like this: imagine you have a collection of cake recipes. You can convert each recipe into a vector embedding, which is like a unique numerical code that represents the recipe’s characteristics (ingredients, cooking methods, flavors, etc.).

Once all the recipes are encoded into embeddings, we can perform a similarity search. This means we can compare the vectors to see how similar the recipes are. For example, the vector for an easy apple pie recipe would be close to the vector for a simple fruit cake recipe because they share similar characteristics (e.g. simplicity, fruitiness). On the other hand, the vector for an extravagant chocolate mousse cake would be farther away because it involves different ingredients and methods.

How to compare vectors?

Vector similarity is a measure of how similar two vectors are (see ep. 4 of ObjectBox Bites). There are three ways to compare vectors: Jaccard Similarity, Cosine Similarity, and L2 Distance (also known as Euclidean distance). Jaccard Similarity calculates the ratio of elements that are common to both vectors divided by the total number of elements in both vectors. Cosine Similarity calculates the cosine of the angle between two vectors. The last method is the L2 distance. It calculates the straight-line distance between two points in space represented by the vectors. This is the most frequently used method in AI applications. It is important to note that the choice of vector comparison method does not affect the mechanics of similarity search.

What is a vector database and how is it related to vector search?

A vector database is a specialized database designed to store, manage, and search vectors efficiently. This efficiency is crucial for handling large datasets and performing fast vector similarity searches. Also, with a vector database, the knowledge of AI models can be improved, adapted, and updated. Therefore, today, most AI apps use a vector database.

Imagine having an AI that knows your habits, your preferences, your health data, maybe even what’s in your fridge, and can use this knowledge to suggest recipes that fit your lifestyle and individual preferences. A standard AI model doesn’t have that data and wouldn’t learn that way, but with a vector database it can. Now, when you search for a “fruit cake recipe”, using this data, it can suggest a “simple fruit cake” without sugar if you usually prefer quick, easy, and healthy recipes, or a “fancy plum cake” if you enjoy more challenging baking projects and don’t like apples. Or, a vegan option, if you have neither milk nor eggs left in the fridge.

This technique is called Retrieval-Augmented Generation (RAG). It enhances the capabilities of LLMs with additional data (e.g. personal data, company data, fresh data) stored in a vector database.

When you query a vector database, it uses the query’s vector representation to find the nearest neighbors in the database.

Nearest Neighbor Search

How do we find the nearest neighbor to our query vector? The most straightforward approach is a brute-force search. It calculates the distance between our query vector and all other vectors in the database, one by one. Any metrics discussed in “How to compare vectors” can be used. However, this brute-force approach has a time complexity of O(N*d), where N is the number of vectors and d is the dimensionality. This becomes computationally expensive for large datasets.

Since exact nearest neighbor search can be slow for massive datasets, we often turn to approximate nearest neighbor (ANN) algorithms. These algorithms prioritize efficiency by finding neighbors that are very close (but not necessarily the absolute closest) to the query vector, significantly reducing search time. 

Continuing with the cooking assistant app example, imagine you’re searching for a “fruit cake recipe”. Assume that in our database, the real closest recipe is “simple apple pie”. With a massive database, an exact nearest neighbor search might take a long time to find the perfect match. However, an ANN algorithm can quickly find a recipe that is very similar to what you’re looking for, such as a “simple fruit cake” or a “basic apple pie”, even if it might not be the exact closest match. This efficiency ensures you get relevant and useful recipe suggestions promptly, enhancing your overall experience without a noticeable compromise in quality.

Approximate Nearest Neighbour Search

Now, let’s delve into the world of Approximate Nearest Neighbor (ANN) algorithms. The way you search for nearest neighbors depends on how the data is stored in the vector database. One of the earliest ANN algorithms, established in 1975, is called k-d trees. These trees work by recursively splitting the data space using hyperplanes, making the search process more efficient (see ep. 5 of ObjectBox Bites). However, k-d trees, like many exact nearest neighbor algorithms, suffer from the dimensionality curse. This means that as the number of dimensions (features) in your data increases, the distance between points becomes less meaningful, making searching very slow in high-dimensional spaces like those used in vector databases. 

For instance, consider simple fruit recipes. With a few features, such as cooking time and number of ingredients, finding similar recipes would be relatively straightforward. However, if we also include many other features like sweetness level, calorie count, fruit type, all specific ingredients, preparation complexity, and user ratings, the number of dimensions increases significantly. In such high-dimensional spaces, the traditional k-d tree method becomes inefficient because the distances between points (recipes) become less distinct and meaningful.

To overcome this challenge, ANN algorithms leverage two main approaches: indexing methods and sketching methods. Indexing methods work by creating a hierarchical data structure that allows for faster exploration of the search space. Imagine a well-organized library with categorized sections instead of just randomly placed books.  Sketching methods, on the other hand,  don’t search the entire dataset directly.  Instead, they create compressed versions (sketches) of the data that are faster to compare with the query vector. This reduces the search time significantly. Often, these two approaches are combined for optimal performance.


A popular example of an ANN search implementation for high-dimensional data is the Hierarchical Navigable Small World (HNSW) algorithm (e.g. implemented in Azure AI). HNSW relies on graph-based indexing to efficiently navigate the data space and find nearest neighbors. For more details watch episodes 6, 7, and 8 of ObjectBox Bites miniseries, where we describe the fundamentals of HNSW.

Take-away notes

To sum up, vector search offers a significant leap forward in how we search for information. By understanding the meaning and relationships behind data, it delivers more relevant and accurate results, even for unstructured data and complex queries. This technology has the potential to revolutionize various fields, from enhancing search engines to empowering AI applications. As vector search continues to evolve, we can expect even more exciting possibilities for navigating the ever-growing ocean of information and unlocking its full potential. This includes operating with data directly on the devices it was created on, reducing cloud costs, eliminating the reliance on an internet connection, and opening up using your private data without it ever being shared (100% private). If you’re interested in other AI and vector database-related topics, check out the ObjectBox mini-series. Stay tuned for more articles in the future.