Vector databases – a look at the AI database market with a comprehensive comparison matrix

by Vivien | May 30, 2023 | AI, Open Source, vector database

Vector databases - a look at the AI database market

⭐ What are vector databases? ⭐ What do you need them for? ⭐ Who is in the market? (Updated Oct 2024)

Includes a comparison matrix of vector database options like Pinecone, Milvus, Vespa, Vald, Chroma, Marqo AI, Weaviate, and Qdrant

In 2023 we saw record fundings of vector database players vector database. Since then almost every general purpose database (like MongoDB, elastic, Orcale MySQL etc.) have added a Vector Search and related features, basically making all of the vector databases too. There is an ongoing discussion if pure players are superior, but as always, the right answer is: “it depends”. Any ways, the vector database market is stilly very hot in Q4 of 2024 🔥

Of course, everyone, not just investors, is interested in the booming AI market. While AI applications have dominated the news for quite some time, the infrastructure software that supports these applications, such as vector databases, has finally gained more spotlight. In the following, we’ll have a look at why vector databases are gaining attention and compare current vector database alternatives.

What is a vector database?

A vector database stores vectors, or more precisely vector embeddings. A vector database therefore is a specialised type of database designed to store and manage large sets of vectors efficiently. However, the challenge and value are not derived from simply being able to store vectors. The value is created by the type of computations that can be run over the stored vector data and the speed with which these computations can be run, e.g. similarity searches.

Vector databases are essentially an important piece of the AI tech stack. They can be used e.g. to give LLMs (Large Language Models) – or more broadly speaking, AI applications – a long-term memory and faster search and querying capabilities. Another important use case is RAG (Retrieval-Augmented Generation).

To give some context: The most traditional databases, SQL databases, store data in rows and columns; graph databases store graphs and object databases store objects.

Because Large Language Models and AI applications rely on vector embeddings, vector databases are especially apt at supporting AI applications.

Accordingly, vector databases are becoming a critical layer in the AI tech stack; they are sometimes also called “AI databases”. However, databases tend to converge over time, meaning that many databases support several different database models.

What is a vector embedding?

A vector embedding is a list of numbers that represent objects and relationships, allowing unstructured data (such as images) to be searched and used. Typically, Large Language Models (more precisely the underlying Machine Learning (ML) algorithms) are used to create these vectors. The ML algorithms analyse large amounts of data to learn how to represent complex / unstructured data in a lower dimensional space (as vectors).

What do vector databases have to do with nearest neighbour search?

Searchability (making unstructured data usable) is at the heart of this concept. The nearest neighbour search is therefore a key concept in vector databases. The distance between vector embeddings expresses the similarity of the vectors (and thus the represented objects). Therefore, as you are searching for the most similar data, the so-called “nearest neighbour search” is a key concept and the time required to find the nearest neighbours is essential.

Do we need special vector databases?

There is already a discussion going on about whether special vector databases are needed or do not warrant a new category in the database landscape. Instead, vector extensions of traditional databases could be supporting the AI market. Both are reasonable expectations, and time will tell. Notable databases that have already added a vector extension include e.g. redis and elasticsearch. Additionally, more and more databases now allow storing vector types.

How does the vector database landscape look like?

To have a look at the current market situation, we are comparing the choices with the most traction, but excluding established players that have added vector capabilities to their existing database offering. Generally speaking we see a lot of very young companies, some companies that did pivot from their original specialization, and massive fundings. Please note: the table is not optimized to be readable on mobile or small screens (there just is a trade-off between providing the information and making it readable on every device).

If you’re on mobile, use this link to view a version that is readable on mobile.

Name

Open Source

License

GitHub stars

Developed in (language)

Summary

Business Model

Embeds / Uses

founding date / first released date

In-memory Unterstützung

Sharding

Index Types

Consistency Model

Benchmarks (Performance?)

Approximate Nearest Neighbor (ANN) Vector Databases

Funding

Who's behind it

HQ in

ObjectBox

Apache-2.0

C++, supports native language APIs in Java, Flutter / Dart, Swift, Python, GoLang, and C++

ObjectBox is an on-device vector database for Edge AI on Mobile, IoT, Embedded and other commodity devices

Free to use; paid Data Sync

HNSW built and optimized from scratch for efficiency / speed on devices with limited resources

development of the initial on-device database started in 2015; released the vector search to become the first on-device vector database for productive use early in 2024

HNSW

Transactionally safe, ACID

Seed in 2018

ObjectBox

🇪🇺

Marqo AI

Apache-2.0

2.8k ⭐

Python

A tensor-based cloud-native commercial Open Source search and analytics engine.

Open SaaS

Tensor-based

❔

HNSW

undisclosed preseed in May 2022

S2Search Australia Pty Ltd

🇦🇺

Weaviate

BSD

5.6k ⭐

Assembly, C++, GoLang

Weaviate is a commercial Open Source cloud-native vector database that stores both objects and vectors.

Open SaaS

❔

started in 2018 as a traditional graph database, first released in 2019

Y, static sharding

a custom HNSW PQ algorithm that supports CRUD

Eventual Consistency

not comparative, just evaluating their own performance

Y (multiple ANN algorithms as long as they support full CRUD)

67.7M USD, series B

SeMI Technologies

🇪🇺

Chroma

Apache-2.0

4.4k ⭐

Python & Typescript

Chroma is a Commercial Open Source vector database

Preparing a (Partly Open) SaaS model* [Commercial Open Source]

HNSW lib, DuckDB; based on ClickHouse

looks like 2022

Dynamic segment placement

20.3M USD, seed

Chroma Inc.

🇺🇸

Qdrant

Apache-2.0

6.6k ⭐

Rust

Qdrant is a Commercial Open Source vector similarity search engine and vector database

Open SaaS

RocksDB

first released: 2021

Y, static sharding

HNSW (SQ & PQ)

Eventual Consistency, tunable consistency

compares to weaviate, milvus, elastic (note: redis took too long to complete)

9.8M €

Qdrant Solutions GmbH

🇪🇺

Milvus

Apache-2.0

18k ⭐

GoLang & Python

Milvus is a cloud-native Commercial Open Source vector database

(Partly Open) SaaS* [Commercial Open Source]

Initial blog post from them said SQLite, but meanwhile they said RocksDB - exchanged?
they also have a ChatGPT-Cache that is build on SQLite
and say "Milvus uses SQLite or MySQL to manage metadata"

founded 2017, first released: 2019

Dynamic segment placement

ANNOY; HNSW; IVF_PQ; IVF_SQ(; IVF_FLAT; FLAT; IVF_SQ8_H; RNSG

Strong, bounded staleness, session, and eventually. The default consistency level in Milvus is bounded staleness.

not comparative

113M USD, series B

Zilliz

🇺🇸

Vespa

Apache-2.0

4.4k ⭐

Java & C++

Vespa is a Commercial Open Source vector database by Yahoo! It is a search engine which supports vector search, lexical search, and search in structured data

Open SaaS

❔

Originally a web search engine (alltheweb), acquired by Yahoo! in 2003 and later open sourced as Vespa in 2017; sinde Oct 2023 spinoff, raised series A in Nov 2023

maintains disk and memory structures for documents

Custom HNSW (Multi-vector hybrid HNSW-IF)

Eventual Consistency

not comparative

Spinoff from Yahoo! in Oct 2023, then raised a 31M USD series A

Yahoo!

🇺🇸

Vald

Apache-2.0

1.2k ⭐

GoLang

Vald is a cloud-native Open Source distributed approximate nearest neighbor (ANN) dense vector search engine

Community project, currently looks like no commercial interests are pursued

uses the vector search engine NGT

Technology incubation at Yahoo! Japan Corporation, development was stared in 2019

❔

N/A

not comparitive, but Vald performance only

Y (NGT)

Yusuke Kato (Yahoo Japan Corporation), Kiichiro Yukawa (Yahoo Japan Corporation)

🇯🇵

Pinecone

Proprietary

Pinecone is a fully managed vector database that specializes in enabling semantic search capabilities

SaaS

built on top of Faiss

first released in 2019

proprietary

Eventual Consistency

more programming language comparison for vector databases

Y (proprietary), plus KNN (with Faiss)

138M, series B

Pinecone Systems Inc

🇺🇸

Want to know more about the vector database market?

Here are some more questions answered for anyone interested

What is an "Open SaaS" business model?

Software as a service (SaaS) refers to software that is managed / hosted for the client and is essentially “rented.” The open in Open SaaS refers to the open source software that is being offered as such a service.

This frequently implies that not all code is open source, particularly that which is part of the managed service / hosting and associated value-adding features. Note: The open source software offered in this manner may or may not be provided by the company providing the software as a service. This has caused some friction in the open source community, as original creators often struggle to make a living, and/or maintainers struggle to keep maintaining the software – while other companies profit. Most famously, huge cloud providers have taken advantage of this option, leading to new licenses that keep the source open but restrict others from hosting as a service without donating the whole source code back to the community.

Why should I care about index types?

Indexes are essentially a way to speed up searching a database. There are several established index types for vector databases and they affect the performance of the database, e.g. the time it takes a query to complete.

What about benchmarks?

You will see, if you review the benchmarks given at the top, that results typically vary. Benchmarks are difficult to do and neutral benchmarks even more so. Certain use cases may favor certain solutions. Therefore, ideally you benchmark based on your specific use case…. but as a first evaluation, try to understand the basic influencing factors and have a look at a handful of benchmarks and explanations. Having said all this: There is a benchmarking tool available for approximate nearest neighbor (ANN) algorithms search. If you use this, you can compare the performance of different databases (with regards to the ANN search) for the same setup, based on the same approach. Also: The underlying libs often used by databases (like NGT and HNSW, see above) have already been benchmarked with it and you can compare to these directly.

Why is the market so hot, how can companies raise so much money?

AI is hot, everyone agrees that data and its management will be key to future success, and the database market is interesting: It is a long established market with many players, yet still demonstrating continually good growth (e.g. 17% in 2020). And the database market history shows that from time to time a new type of database comes up, and with it, the creation of a new market category. In such a market, typically the market creator “takes all” (not quite literally, but such a significant share, definetely the vast majority, that all other players are not attractive from a VC-perspective). Such a market could easily be worth 100M+ in ARR. Examples from the last 20 years: MongoDB (NoSQL databases), Cockroach (NewSQL databases), Neo4J (Graph databases), Influx (Time-Series databases). So, VCs are looking to find the next new type of database that can create a market… Maybe it will be vector databases? However, the database market has also shown to take 10 years+ for players to become profitable, so expect a longterm game. The race is still on for Edge Databases we think 🙂

Want to know more about the database market?

We recommend checking out db-engines. The website compares all relevant systems and has tons of data from the last 20 years. Note: They do only add databases once they have some traction and notability, not any hobby project. Accordingly not all databases of the above comparison have been added to the website yet.

What is an Edge Database, and why do you need one?

by Vivien | May 30, 2023 | Edge Computing, Insights, IoT, Mobile Database

Edge Databases – from trends to use cases

Data is decentralized. Cloud computing is centralized. Forcing the decentralized world into the centralized cloud topology is not only inefficient, but also economically, ecologically and socially wasteful – and sometimes simply impossible.

To drive digitization and extract value from decentralized data, we need to give the cloud an edge, or more precisely add Edge Computing. Edge computing is a decentralized topology for storing and processing data as close as possible to the data source, i.e., the place where the data is produced, at the edge of the network.

Valuable data is increasingly generated in a decentralized manner – outside traditional and centralized data centers and cloud environments. The dominance of centralized cloud computing approaches slows down digitization and the use of this existing decentralized data. Therefore, according to Gartner (2023) “Edge computing is integral to digital transformation”, and we need infrastructure technologies for the edge that enable developers to quickly and reliably work with decentralized edge data.

Edge Database (Foundation for Edge Data Management) is a new type of database that addresses these requirements. Developers need fast local data persistence and decentralized data flows (Data Sync) to implement edge solutions. Edge Databases solve these core edge functionalities out-of-the-box, allowing application developers to quickly implement edge solutions.

Megatrend to decentralized Edge Computing
Urgently needed: Software infrastructure for Edge Computing
What is an Edge Database?
When do you need an Edge Database?
Edge Database Use Case Example in Manufacturing
The Future of Edge Databases

Megatrend to decentralized Edge Computing

By 2030, 30+ billion IoT devices will be creating ~4.6 trillion GB of data per day. The growing numbers of devices and data volume, variety, and velocity, as well as bandwidth infrastructure limitations, make it infeasible to store and process all data in a centralized cloud. On top, new use cases come with new requirements, a centralized cloud infrastructure cannot meet. For example, soft and hard response rate requirements, offline-functionality, and security and data protection regulations.

These trends accelerate the shift away from centralized cloud computing to a decentralized edge computing topology. Edge computing refers to decentralized data processing at the “edge” of the network. For example, in a car, on a machine, on a smartphone, or in a building. Hardware specifications do not capture the definition of an “edge device”. The crucial point is rather the decentralized use of data at, or as close as possible to, the data source.

Edge computing itself is not a technology but a topology, and according to McKinsey, one of the top growing trends in tech in 2021. The technologies needed to implement the edge computing topology are still inadequate. More specifically, there is a gap in basic “core” edge technologies, so-called “software infrastructure”. This gap is one of the main reasons for the failure of edge projects.

Needed: Infrastructure Software for Edge Computing

With computing shifting to the edge of the network, the needs of this decentralized topology become clear:

Need for fast local data storage

→ i.e. a machine on the factory floor collects data on stiffness, friction, pressure points. There is limited space on the device, and typically no connection to the Internet. Even with an Internet connection, high data rates quickly push the available bandwidth, as well as associated networking / cloud costs, to the limit. To be able to use this data, it must be persisted in a structured manner at the edge, e.g. stored locally in a database.

Need for reliable on-device data flows

→ i.e. the car is an edge device consisting of many control units. Therefore, data must be stored on multiple control units. In order to access and use the data within several of the control units of the car, the data must be selectively synchronized between the devices. A centralized structure and thus a single point of failure is unthinkable.

Need for edge-to-edge-to-cloud data flows

→ i.e. in a manufacturing hall: Typically, you will find any number of diverse devices from sensors to brownfield to greenfield devices, and no internet connectivity. At the same time, there are diverse employee devices such as tablets or smartphones, as well as central PCs, and a cloud. To extract value from the data, it must be available in raw, aggregated, or summary form, in different places. This means it needs to be synchronized efficiently and selectively, with possible conflicts resolved.

Need for flexible edge data management

→ e.g. with the rise of IoT, time-series data have become common. However, time series data alone is usually not sufficient, and needs to be combined with other data structures (like objects) to add value. At the same time, a push to standardize data formats in industries (e.g. VSS in automotive or Umati in Industrial IoT) requires that the database supports flexible data structures.

Developing solutions without software infrastructure on an individual level is possible, but has many drawbacks:

Custom in-house implementations are cumbersome, slow, costly, and typically scale poorly. Oftentimes, applications or certain feature sets become unfeasible to deliver because of the lack of core software infrastructure. Legacy code and individual workarounds create problems over the lifetime of a product. Instead of a thriving ecosystem, only a few big players are able to implement edge solutions. Innovation and creativity are limited. An edge database is part of the solution and enables the entire edge ecosystem to build edge applications faster, cheaper and more efficiently.

What is an Edge Database?

An Edge Database is a type of database specifically tailored to the unique requirements of the Edge Computing topology. Edge Databases run directly on-device, locally, and make it easy for app developers to access decentralized data from edge devices when and where needed. Using an Edge Database removes the burden of implementing ways to synchronize data, which is non-trivial, time-consuming, risky, and brings ongoing maintenance needs. Let’s look at this in more detail:

First, an Edge Database is optimized for resource efficiency (CPU, memory, …) and performance on resource-constrained devices (embedded devices, IoT, mobile). It has a small footprint of a few megabytes max. Traditional databases such as MySQL or MongoDB are too large and slow for typical edge devices, making them unsuitable for computing at the edge. Nevertheless, with integrations like the one between ObjectBox and MongoDB, developers can now combine ObjectBox’s on-device efficiency and offline-first capabilities of Edge Databases with MongoDB’s scalable cloud platform to enable seamless, bi-directional synchronization between the edge and the cloud.

An edge device without data flows to/from other devices is just a data island with very limited utility. Accordingly, an Edge Database must support the management of decentralized data flows. There is no more efficient way than at the database level. This ideally includes a range of conflict resolution strategies due to the decentralized and multi-directional structure of the Edge.

Last not least, data security is of growing importance and data in motion needs to be protected. Data at rest is on a database level often protected by the OS and therefore less of a concern for most applications.

When do you need an Edge Database?

Most IoT applications need to store and synchronize data. An Edge Database is always useful when functions / applications are planned that:

should work offline and independent of an internet connection
need to guarantee fast response times
work with a lot of, possibly high-frequency data
need to serve many devices at the same time
need historical data

In addition, developers also often decide to use an Edge Database to save time and nerves, or to be able to react quickly and flexibly to future requirements.

Edge Database Use Case Example in Manufacturing

Today, you can find everything from low-frequency brownfield devices to high-frequency greenfield devices on a factory floor. As a rule, the machine controllers in use are not designed to store or transmit data. They usually lack not only the functionality, but also the resources to support this. Therefore, additional edge devices are often needed to collect, analyze and interpret the huge amounts of data that each machine produces on site. For such an edge device, rapid data persistence and ingestion, and efficient data flow from edge-to-edge and edge-to-cloud are at the heart of value creation. The clear separation of machine control and edge data processing unit ensures that there is no risk of unintentional interference with the machine controller. An edge device with a powerful edge database can support multiple use cases on the shop floor today:

1. Operational efficiency

Process optimization along the line to increase quality and reduce damage. When the first machine in a production line uses a new batch of material, i.e. in sheet metal processing, one of the first steps is to cut a sheet to the required size. At this stage, the machine can already detect the differences in the metal compared to a previous batch (deviations are allowed within the DIN standard). With an Edge device this data can be evaluated, and the relevant information passed on to the next machine. With this data machines further down the line can avoid damage / breakpoints of the material.

2. Condition monitoring

Continuous machine condition monitoring reduces downtime and increases maintenance efficiency. A constant stream of high-frequency machine data is compared against the fingerprint of the machine. Any slight deviation is immediately detected and reported. Catching deviations early reduces down-times and costly repairs.

3. Historical Data

Historical data is stored for learning and training to optimize the production line. With an Edge Database, the data is persisted and thus available in the event of faulty behavior. In case of an error, the data preceding the incident can be analyzed and used to find the causes and predict, or even avoid, such an error in the future. Chances are that “fuzzy expert knowledge” already available at the production site can be translated into deterministic rules when tested with these data sets.

The future of Edge Databases

Edge computing provides numerous benefits and enables many applications and functionalities that are only possible with edge computing. However, only a few (usually large) players have been able to create value in edge computing projects, gaining competitive advantages. One reason is a lack of basic edge software. A thriving edge ecosystem necessitates edge software infrastructure that addresses the fundamental recurring needs of edge projects. Edge databases are a critical component in the development of such an ecosystem.

Looking ahead, the emergence of on-device vector databases, coupled with small language models (SLMs), is transforming the landscape of AI applications. These technologies enable AI apps to run directly on edge devices, providing long-term memory, improving performance, and significantly reducing resource consumption. By processing data locally, they eliminate the need for constant cloud connectivity, enhancing privacy and efficiency. Companies like Apple have already embraced on-device AI (Apple Intelligence), showcasing its potential to deliver advanced functionalities seamlessly. This shift represents a game-changer, making AI more sustainable, scalable, and integrated into everyday use.

Green Coding: Developing Sustainable Software for a Greener Future

by Vivien | Apr 19, 2023 | Edge Database, Insights

Digitization helps to save CO₂ – many experts agree on that. But things are not that simple, because the creation of software and its use contribute to greenhouse gas emissions too. All code creates a carbon footprint. Software development and use affect the environment from the energy consumed while running to the associated electronic device waste. Choosing a sustainable software architecture matters, but every developer also can make a difference by applying green coding principles.

This article will explore the importance of green software development and its main principles.

1. Green Software Development: Balancing Digitization and Environmental Sustainability

1.1 What does sustainability in software development mean?

1.2 What is green coding / green software development?

1.3 What is the difference between Green IT and Green Coding?

1.4 Why is it time for developers to prioritize environmental sustainability?

2. How to put sustainable software development into practice

2.1 Guidelines to making your code more sustainable

2.2 Examples of sustainable coding: More impactful than you would expect

3. Sustainable Edge Data Managment w. ObjectBox – a ready-made developer tool

4. References

Green Software Development: Balancing Digitization and Environmental Sustainability

In this section, we’ll first define some important terms in the topic of environmentally conscious software development. Then, we’ll discuss why it is relevant and discussing the broader benefits of adopting green coding practices.

What does sustainability in software development mean?

In our view, sustainability in software development (also “green software development”) entails developing and maintaining software in a way that is not only environmentally, but also socially and economically responsible. So, what really counts is the long-term bottom-line value from a general societal perspective, not an “individual balance sheet”.

There are many trade-offs in such an ambition, and therefore sustainable software development is rather a set of guiding principles than hands-on measures that are truly the same for everyone. Let’s dive a bit into how sustainable software development can contribute to all three aspects:

Environmental aspects

Since software is a significant source of direct greenhouse gas emissions, it is becoming more important to create software that reduces resource use as much as possible. As the world becomes more reliant on technology, energy consumption and carbon footprint of software will continue to grow. By adopting green software development practices, software developers can help to mitigate these environmental impacts.

Broader Economic contribution

If a software uses less energy and resources to accomplish the same tasks as another software, the users of that software can reduce their operating costs and improve their bottom line. Increasing the longevity of hardware (less wear, but also less hw requirements extending the usability of existing hw) also yields direct economic savings for the software users (companies as well as individuals). On a broader level, this compounds significantly over the number of users and with time and thus contributes to economic welfare. What sounds like a small contribution does add up tremendously in the end…

Social impact

Sustainable software development includes responsibility for the social impact of the software created. As a result, sustainable software aims to be transparent, inclusive, and offer data sovereignty. By giving individuals and organizations greater control over their own data, software empowers them and protects their privacy. At the same time, it promotes greater accountability and transparency in data-driven decision-making.

Overall, sustainability in software development involves taking a holistic approach. On top, sustainable software companies take steps to minimize negative impacts and promote positive ones over the long term.

This is why it has been one of our core values since we started ObjectBox:

Be Sustainable in every respect – we apply sutainability to our technology, as well as the people and small every-day decisions. ObjectBox aims to be the most resourceful data management solution for connected devices. We strive to save resources (energy, CO₂, bandwidth, time, etc.), but also always choose the sustainable path (recycled paper, saving energy, etc.), and support our employees to lead balanced and sustainable lives.

What is green coding / green software development?

Recently, the term “green coding” has emerged to describe the practice of creating and writing code (aka software) in a way that minimizes its environmental impact. This can involve using efficient code that consumes less energy, optimizing data usage, and reducing electronic waste.

What is the difference between Green IT and Green Coding?

Green IT is primarily about the hardware and the optimization of data centers. Today, it often actually is about optimizing cloud usage. The code decides whether this hardware is used efficiently. By contrast, green coding is about making the code more efficient, so that running the code (e.g. using an app on the smartphone, or using an email program) uses less resources and less electricity, thus producing less CO₂.

Why is it time for developers to prioritize environmental sustainability?

Various studies estimate the Carbon footprint of the digital economy to be between 2.3 – 3.7% percent of global CO₂ emissions 😱 [1]. Although the impact of software on the environment may not yet be as dramatic as that of manufacturing, it keeps growing rapidly each year. By taking sustainable decisions in software development, we can make it part of the carbon solution of the future.

Every line of code – scaled up to hundreds, thousands, or even millions of devices (desktops, smartphones, tablets…) worldwide – has the potential to significantly reduce energy consumption and CO₂ emissions.

How to put sustainable software development into practice?

We believe two key aspect to develop sustainable software, that creates bottom-line value, are:

minimize the resource consumption of software especially during operation, where most resources are consumed – be dilligent about that; it compounds
keep data as much as possible where it is produced, used and belongs (e.g. with the end users) and avoid unnecessary data transferals, superfluous cloud use, and unnecessarily storing data in the cloud

Both measures have significant environmental, social, and economic impact, short- and long-term.

It’s time we as developers start thinking about our impact on the planet and make sustainability a part of our everyday coding mindset. We can make a difference by incorporating sustainability into every action and decision we take when developing software. Careful measuring and optimizing the resource along the way is also important. The welcome side effect: fast software that is cheap to run and fun to use 🙂

For example, at ObjectBox, we’re all about maximizing the use of computing resources and minimizing resource waste of every line of code (LOC). This makes ObjectBox not only environmentally sustainable, but at the same time superfast, usable on low end devices w. little hw requirements, and cheap in operational costs 🤯

💚 Responsible development practices pay off in several respects and we really cannot see a huge tradeoff. All it costs is spending more time and brain on optimizations, benchmarking, and dilligently applying this approach to every line of code.

💚 As a developer tool, our impact is broader than a developer’s impact on end-users. So, we’re committed to using resources efficiently and reducing waste at every stage of the game.

Guidelines to start making your code more sustainable

Some more tipps how to put sustainable software development into practice:

Energy efficiency: Developing software that is energy-efficient can help to reduce its environmental impact by minimizing the amount of energy required to run software.
Responsible sourcing: Using responsibly sourced hardware, software, and other materials can help to reduce the environmental impact of software development.
Longevity: Developing software that is designed to last can help to reduce waste and promote sustainability by reducing the need for frequent updates and replacements.
Accessibility: Making software accessible to a wide range of users can help to promote social sustainability by ensuring that everyone has access to the benefits of technology.
Data sovereignty, privacy and security: Protecting user data and maintaining strong cybersecurity measures can help to promote sustainability by preventing data breaches and other security incidents that can have negative social and economic impacts.

Examples of sustainable coding: More impactful than you would expect

1. How can a millisecond be worth 2 days?

Real world example: By reducing the resolution of images in a banking app with 500.000 users, whose users on average opened it daily, developers saved more than 2 days of total operational time (up time) [2].

2. How can 2 grams of CO₂ savings / hour be worth 330.000 t CO2?

Theoretical consideration: Netflix states that streaming its content produces 55 grams of CO₂ per hour [3]. This gives us 40 kilograms of CO₂ per year for daily streaming of two hours per person [4]. With Netflix users being 230M, a reduction would have an enormous scaling factor [5]. Assuming a Netflix developer reduces the 55 grams to 53 grams, you get 330 kt of CO₂ in potential savings. Note: This is a highly theoretical example, just to demonstrate the thinking.
Anyways: Individuals can’t save that much as easily. That’s the impact you as a programmer have!

3. How much CO₂ can local storage save in 1 million cars?

Sending and storing 1 GB of data in the cloud needs about 5 kWh of electricity, while local storage only needs about 0.000005 kWh, which is a million times lower. Making the switch to local storage in 1 Million cars would lead to saving 905 kg of CO₂ every second. If you want to know what that actually means, you can translate that into equivalents: CO2 equivalencies or the CO2 calculator

👉 These examples clearly illustrate the potential impact of shifting towards an environmentally conscious mindset when developing software. Now that we know the why, it’s time to discuss the how.

Sustainable Edge Data Managment w. ObjectBox – a ready-made developer tool

ObjectBox is a free Edge Database that can help reduce the environmental impact of apps. It is optimized for computing resource efficiency and empowers developers to store and use data locally and create offline-first apps. Unless the data is really needed in the cloud, this is way more energy-efficient and sustainable compared to a cloud setup. On top, it works independant from an Internet connection being available and is superfast while saving battery, making it an ideal choice for apps that prioritize sustainability.

What is an Edge Database?

An Edge Database is a type of database that is used on the “edge” of a network, closer to the data sources and devices generating data. Traditional databases, on the other hand, are usually set up in centralized data centers or in the cloud.

Edge databases are essential when devices need to work offline, guarantee response times, speed is of the essence, you have limited Internet connectivity, mission-critical scenarios, or when handling high-frequency data. By processing data locally on the edge, Edge Databases can reduce latency and improve performance while also reducing the amount of data transferred over the network.

Edge databases have a small footprint and are designed to run on restricted devices such as routers, IoT gateways, mobile phones, and other embedded systems. They typically incorporate features needed in distributed systems, such as data synchronization, caching, and offline support to ensure that data remains available even in the event of network outages or other disruptions.

ObjectBox Sync is a highly efficient and sustainable data synchronization solution. It reduces the amount of energy used by having as little overhead as possible when sending data combined with solid compression, avoiding data transformations, and only syncing data changes instead of sending all data to the cloud all the time. Developers have control over what data is synced when.

Overall, ObjectBox DB + Sync is a powerful tool for building fast apps that prioritize consuming less energy and saving device resources. By storing data locally and only syncing when and where needed, developers can ensure that their apps are as sustainable as possible, and save on cloud costs along the way.

References

1. https://theshiftproject.org/en/article/lean-ict-our-new-report/

2. https://www.gft.com/00de/de/technology/greencoding

3. Netflix Environmental Social Governance Report 2021

4. https://variety.com/2019/tv/global/netflix-cindy-holland-subscribers-watch-average-two-hours-day-1203159868/amp/

5. https://www.statista.com/statistics/250934/quarterly-number-of-netflix-streaming-subscribers-worldwide/

6. https://www.grcooling.com/the-plane-truth-about-environmental-sustainability/

What is Data Synchronization + How to Keep Data in Sync

by Vivien | Dec 29, 2021 | Edge Computing, Insights, IoT, Sync

What is Data Sync / Data Synchronization in app development?

Data Synchronization (Sync) is the process of establishing consistency and consolidation of data between different devices. It is fundamental to most IT solutions, especially in IoT and Mobile. Data Sync entails the continuous harmonization of data over time and typically is a complex, non-trivial process. Even corporates struggle with its implementation and had to roll back Data Sync solutions due to technical challenges.

The question Data Sync answers is

phone-data-sync-with-machine-payment-automatic-data

How do you keep data sets from two (or more) data stores / databases – separated by space and time – mirrored with one another as closely as possible, in the most efficient way?

Data Sync challenges include asynchrony, conflicts, slow bandwidth, flaky networks, third-party applications, and file systems that have different semantics.

Data Sync versus Data Replication in Databases

Data replication is the process of storing the same data in several locations to prevent data loss and improve data availability and accessibility. Typically, data replication means that all data is fully mirrored / backed up / replicated on another instance (device/server). This way, all data is stored at least twice. Replication typically works in one direction only (unidirectional); there is no additional logic to it and no possibility of conflicts.

In contrast, Data Sync typically relates to a subset of the data (selection) and works in two directions (bi-directional). This adds a layer of complexity, because now conflicts can arise. Of course, if you select all data for synchronisation into one direction, it will yield the same result as replication. However, replication cannot replace synchronization.

Why do you need to keep data in sync?

Think about it – if clocks were not in sync, everyone would live on a different time. While I can see an upside to this, it would result in many inefficiencies as you could not rely on schedules. When business data is not in sync (up-to-date everywhere), it harms the efficiency of the organization due to:

Isolated data silos
Conflicting data / information states
Duplicate data / double effort
Outdated information states / incorrect data

In the end, the members of such an organization would not be able to communicate and collaborate efficiently with each other. They would instead be spending a lot of time on unnecessary work and “conflict resolution”. On top, management would miss an accurate overview and data-driven insights to prioritize and steer the company. The underlying mechanism that keeps data up-to-date across devices is a technical process called data synchronization (Sync). And while we expect these processes to “just work”, someone needs to implement and maintain them, which is a non-trivial task.

Growing data masses and shifts in data privacy requirements call for sensible usage of network bandwidth and the cloud. Edge computing with selective data synchronization is an effective way to manage which data is sent to the cloud, and which data stays on the device. Keeping data on the edge and synchronizing selective data sets effectively, reduces the data volume that is transferred via the network and stored in the cloud. Accordingly, this means lower mobile networking and cloud costs. On top, it also enables higher data security and data privacy, because it makes it easy to store personal and private data with the user. When data stays with the user, data ownership is clear too.

Unidirectional Data Replication

Bidirectional Data Synchronization

Out-of-the-box Sync magic: Syncing is hard

Almost every Mobile or IoT application needs to sync data, so every developer is aware of the basic concept and challenges. This is why many experienced developers appreciate out-of-the-box solutions. While JSON / REST offers a great concept to transfer data, there is more to Data Sync than what it looks like at a glance. Of course, the complexity of Sync varies widely depending on the use case. For example, the amount of data, data changes, synchronous / asynchronous sync, and number of devices (connections), and what kind of client-server or peer-to-peer setup is needed, all affect the complexity.

What looks easy in practice hides a complex bit of coding and opens a can of worms for testing. For an application to work seamlessly across devices – independent of the network, which can be offline, flaky, or only occasionally connected – an app developer must anticipate and handle a host of local and network failures to ensure data consistency. Moreover, for devices with restricted memory, battery and/or CPU resources (i.e. Mobile and IoT devices), resource sensitivity is also essential. Data storage and synchronization solutions must be both effective / efficient, and sustainable.

How to Keep Data in Sync Without the Headache?

Thankfully, there are out-of-the-box data synchronization solutions available on the market, which solve data syncing for developers. They fall broadly into two categories: cloud-dependent data synchronization, and independent, “edge” data synchronization. Cloud-based solutions, like Firebase, require a connection to the internet to function. Data is sent to and requested from the cloud constantly. Edge solutions, like ObjectBox, also offer “Offline Sync”: Data is stored in an efficient on-device database, synchronization on and between edge devices can be done continually without an Internet connection, and Dat Sync with a cloud or a backend that is not located on premise occurs once the device(s) goes online. Below, we summarize the most popular market offerings for data synchronization (offline and cloud based):

Couchbase

Couchbase is a Cloud DB, Edge DB and Sync offering that requires the use of Couchbase servers.

Firebase

Firebase is a Backend as a Service (BaaS) offering from Google (acquired). Google offers it as a cloud hosted solution for mobile developers.

Mongo Realm

Realm was acquired by MongoDB in 2019; the Mongo Realm Sync solution (Atlas Device Sync) used Realm DB on edge devices and synchronized with a MongoDB hosted in the cloud. However, MongoDB recently announced end-of-life for it.

ObjectBox

ObjectBox is a DB for any device, from restricted edge devices to servers, and offers an out-of-the-box Sync solution. ObjectBox enables self-hosting on-premise / in the cloud, as well as Offline Sync.

Parse

Parse is a BaaS offering that Facebook acquired and shut down. Facebook open sourced the code. The GitHub repository is not officially maintained. You can host Parse yourself or use a Parse hosting service.

Data Sync, Edge Computing, and the Future of Data

There is a megashift happening in computing from centralized cloud computing to Edge Computing. Edge computing is a decentralized topology entailing storing and using data as close to the source of the data as possible, i.e. directly on edge devices. Accordingly, the market is growing rapidly with projections estimating continuing growth with a 34% CAGR for the next five years. The move from the cloud to the edge is strongly driven by new use cases and growing data masses. Edge data persistence and Data Sync (managing decentralized data flows), especially “Offline Sync”, are the key technologies needed for Edge Computing. Using edge data persistence, data can be stored and processed on the edge. This means application always work, independent from a network connection, offline. Faster response times can be guaranteed. With Offline Sync, data can be synchronized between several edge devices in any location independant from an Internet connection. Once a connection becomes available, selected data can be synchronized with a central server. By exchanging less data with the cloud or a central instance, data synchronization reduces the burden on the network. This brings down mobile network and cloud costs, and reduces the amount of energy used: a win-win-win solution. It also enables data privacy by design.

Try the fastest Edge Sync solution today

Building a Business on Open Source

by Vivien | Jul 1, 2021 | Insights, Open Source, Startup

What is open source software?

For the sake of unambiguity: Open source software (OSS) primarily means that the source code of the software is accessible and users are free to use the code as they please. Depending on the license, you might be expected to attribute the source code to the authors and / or commit code enhancements back. Note: It’s “free” as in “freedom” not as in “free beer”.

Open Source and Commercialisation?

The origins of open source did not entail commercialization thoughts. However, in the last 20 years a lot of things have changed, and open source projects have seen commercial successes – though not always by the creators and maintainers… Open source is in its core tied to a philosophy and value set for many people. Simplified: For the developer community by and large open source is considered to be “good” versus proprietory source code is considered to be “evil”.

In any case, open source is one way to keep up an active vibrant developer ecosystem that empowers individual developers as well as startups and smaller players. Open Source is actually one piece of the IT ecosystem that helps balance the Big Tech and drive overall innovation. However, we also believe the open source ecosystem needs more balance to be successful longterm. If widely used open source repos cannot even sustain the half or full developer resource needed to maintain them, then there might well be a flaw in the system. If startups cannot build a business around their widely used open source code to sustain it longterm, it is to the disadvantage of the community, especially for the individual developers and SMEs. And likely, the learning at some point will be to keep the source closed instead.

In the following we will share, why we believe now is the unique opportunity to add fairness and balance for the value creators to the open source ecosystem to keep that ecosystem thriving and successful longterm.

What do we mean with “building a business on open source”?

In many talks with many people, we found there’s at least two diametric conceptions of building a business on open source:

1) using open source software for free and building something around it to earn money
2) developing a solution and open sourcing it or parts of it as part of the business model

In this article, we mean the latter and it inherently entails contributing a useful part of a solution to open source. For some open source enthusiasts a company needs to open source everything to be an open source company, and that’s ok. It is just our definition for this article.

A look at the market – the struggle of open source businesses

The Open Source Gold Rush: Success Stories

In the last years there have been many open source success stories, e.g. MongoDB, elastic, Cloudera all IPOd very successfully. There seemingly is a lot of money in open source businesses, e.g. a study by Fraunhofer concluded that “the EU economy is hugely benefiting from global OSS.” [1] Also, companies and big corporations are way more open to work with open source software, indeed 2020 was the first year where open source databases were on par with closed-source databases with regards to corporate adoption (see chart). [2]

And a recent (2021) report showed that across 17 industries, from 1,546 codebases 98% contained open source code. [3] There even is a bit of a hype that open source is the path to success. Now that it’s clear that it is possible to build a business with open source software, VCs also are more open to funding open source businesses. An Andreessen Horowitz report reveals that OSS companies have raised over $10B in capital with a trend towards bigger and bigger deals. [4] Annual invested capital in open-source and related dev tools has increased at around 10% CAGR over the last 5 years. [5] In the years 2018 and 2019 acquisitions, mergers, and IPOs from open-source companies generated over 80USD billion liquidity value according to Bessemer Venture Partners. [6]

The struggle of turning Open Source into a Business

Historically, open source companies have struggled with turning open source adoption into monetary success, “less than a decade ago open source was considered almost impossible to monetize.” [7] Sadly, that’s still a reality today for many open source maintainers and companies alike. Lots of open source maintainers with widely used open source code (“successful open source”), cannot get enough financial support to maintain the code. Of course, there are some successes, but in the end that might also be a question of ratios. For example, in 2020 GitHub reported having more than 190 million repositories. Even if only 10% of those do want to build a business on top of their code, how many of those see a financial reward? Gut feel: Far less than typical startup success odds. On top: What looks successful from the outside, might not really be a viable self-sustained business. Despite its many users, MongoDB spent $100M on development, and it took them more than 10 years to become profitable according to their own statements. [8]

A lot of tech companies struggle with – and spend a lot of time on – all the decisions around an open source business model. It isn’t easy, read up how GitLab struggled with finding a business model, or look closer into the MySQL story, and the MariaDB journey (which is a MySQL fork by the founders and original authors of MySQL); look at blog posts from CockroachDB, MongoDB, or elastic on open source – and what you see is a constant re-positioning of open source strategies.

As Mike Volpi from Index Ventures noted at the Index Open Source Summit (2021): “It took Mongo DB 10 years to derive the business model they run now and monetize successfully…” Wow, 10 years to somewhat successful monetization – and that is one of the major open source success stories.

Open sourcing your main technology as a strategy

In this article, we take a deeper look at open source as a pro-active business strategy.

Open Source to Build Traction

Traction is the most obvious reason to open source your product. It works like Freemium in the Mobile Games market – or more generally the Mobile Apps market. It’s a great way to evaluate product-market-fit and build traction. When you have that, you can think about monetization.

However, there is a big difference between giving something away for free and open sourcing it. If we stay in the mobile app world: Would open sourcing the app help with traction? Would it jeopardize the business model? Unless the main target users are developers, at least in the beginning likely not – less than making the app / game available for free in any case. However, once the app grows at amazing pace, open source availability could become a challenge in several respects.

The most obvious would be fast followers entering with that same game and potentially much bigger marketing budgets and better customer access (e.g. on the apps store). Think what would have happened if WhatsApp would have open sourced all its code from day 1 on top of giving the app away for free? It is a legit hyothesis that a fast follower could have scraped some of the market, changing the whole story. On the other hand, if they would open source all their code base now, how much would it harm them? At some point, it beame all about the traction, brand, customer access, so, I would think, it wouldn’t harm them at all at this point. So, driving traction with open source is probably only a viable idea if you address developers or engineers. It’s clearly a phenomenon of the developer-led landscape, and acts as a developer distribution channel. This being said, the price of open source traction is commercialization. It’s a straight forward trade-off: The more open and free your license is, the harder it is to monetize later on.

Open Source to Build Trust

Trust is something that is likely more important for certain software types (e.g. B2B and core tech).

ObjectBox is a database and with that it is a data-centric “core technology” / software infrastructure, sitting at the heart of a company’s solution. Anything that gets used at the heart of other companies or their solutions needs a lot of trust. Trust is easier to come by with size, “no one was ever fired for choosing SAP.” Being a small startup lies at the opposite on that spectrum for many decision makers. Open Source can be a way to overcome this specific challenge and build trust in three ways:

Transparency: The freedom to verify what the code enables; the internal developer team can check the code and vouch for the solution
Risk-reduction: The freedom to change and maintain the code oneself gives independence from the authors and the success of the solution
Quality: If an open source solution is actively used by a large number of developers quality inevitably goes up

So, if you are looking for adoption from big players in heavily regulated or security-concerned industries, e.g. medical, manufacturing, automotive, anything with mission-critical networks, open source can help you overcome many of the adoption hurdles you are facing.

Open Source as an IP Strategy

Seems counter-intuitive, right? Well, if you are not aiming to patent your technology, you still might not want someone else (who has been working on the same problem) to patent the same technology harming your freedom to operate. You can protect yourself from that risk by open sourcing it. This can come in the form of a copyleft license, designed to encourage further innovation advancements to the benefit of all, but also limiting the commercial exploitation opportunities for everyone. Or, you can choose a more permissive license, allowing people with commercial interests to keep any advancements they make to themselves.

Note: Open source code is not a blueprint with exact instructions; there are no obligations to provide clear docs or explanations. While a majority of open source projects strive to deliver a code base that is readable by others, it is not controlled. So, while open sourcing a technology harms patenting it, unfortunately, a way to still protect it, is making it hard to understand. On the other hand, a patent must have an extensive explanation. This makes it easily repeatable by others in the future, after the end of the patent protection, or as a basis for further research (and ways to tweak it in a novel enough way).

Although it often feels like open source is on the other spectrum of patents, a patent has a limited timeframe and people can learn from it even before it expires. The deal is basically an exchange of knowledge (to be used in the future) for protection (for commercially exploiting it). Keeping it a trade secret has other risks, but could mean that an invention wouldn’t be shared with others for a truly long time. And of course the protection encourages big companies to invest big budgets in R&D too. Delayed open source actually has many similarities with a patent, in both cases the tech is only made available for advancements and unrestricted use after a certain time frame has ended.

Open Source for the sake of it

There are a lot of ideas floating around open source, and some pressure from the developer community to open source everything. Among developers, open sourcing is considered to be good, social, fair, transparent, and worthy. While there are many advantages in open source, it has turned into a kind of “political tool”, and that’s a downside – and probably the opposite of the original idea.

Consideration 1: How is a great software supposed to be maintained and advanced without anyone providing funds? When MMOGs (Massive Multiplayer Online Games) became a thing, people understood that there was a constant cost associated with it and were willing to switch from a one-off fee to monthly payments. Software typically needs to be maintained too. So, there are ongoing development costs associated with a piece of software, even if it is not hosted. So, who benefits from open source in the end, if the original creators cannot keep up their work (assuming they need to eat and sleep)? Before pushing everyone to open source, maybe read here, here, here, or here about open source maintainers struggling under the pressure and dealing with burnout. On the flip side, if a company markets itself heavily as an “open source company”, they should give considerable parts of their own value creating solution back to the community. Using open source tools and building on top of open source code (and even committing back to these solutions) does not mean you are an open source company: If you want to reap the marketing benefits of calling yourself an “open source company” then you should truly be one and commit your value back to open source.

Consideration 2: Who benefits if another company pulls the repo, adds “sparkles”, maybe even some “missing features”, or merely a big “brand name”, or the “marketing budget” and makes a ton of money selling the solution? This is of course assuming a permissive license was used. Well, from an open source perspective that is perfectly fine, and part of the intention of open source. So, it’s great, right? We think, it is easy to understand that some authors who have put all their “free time” / unpaid time into that code struggle to accept when this happens, especially if they have a hard time supporting themselves. But we also understand that big companies with investors (stakeholders…) that have invested heavily in R&D and might or might not yet have reached profitability, don’t really like to see this happen. Unless you are really in it for the fun and driven by altruism and will be in perfect harmony with other people using your code to make money, you should look closely if and how you want to open source your code.

Open Source to save development costs

There is the idea floating around that you can develop your project for free using the open source community. We doubt it works out for many. Of course, if Google maintains a repo that is a base technology used by many developers, developers might want to commit something (anything really) for fame, to be part of it, maybe to get noticed. However, the “anything really” is already a problem: Someone needs to review the submission, respond, potentially rework it and so on… Most other repos will probably not get too many commit requests (let alone from the best tech talent around). Even then, onboarding a large community of unknown developers and letting them commit to your code has its challenges – especially if you are quality-conscious and / or trying to build a business. It creates a lot of work to review commits and reject / merge them. And on top of that from a legal perspective you need to have a waterproof contributors license signed by anyone committing. There clearly is some work involved in the process, maybe more than what it is worth sometimes.

Also consider this: Most successful open source projects that turned into a business success have limited contributors and / or only internal (contracted) contributors. For example, SQLite 99% of the code was done by Richard Hipp (author and founder of SQLite), and MongoDB stated that about 98-99% of the code was done internally. Redis was almost exclusively coded by Salvatore Sanfilippo. In a presentation from Index Ventures (one of the most renowned open source VCs), one criteria for potentially successful open source businesses was that at least 90% of the code base was developed internally – and of course that the team owned all the IP. If you are after cheap development and external help with your project, maybe take a closer look if open source is the right path.

What open source business models exist?

The following open source business models are common, but typically used in combination and not as pure models, e.g. most open source companies offer paid support, but rarely only paid support. Note: With time the examples may become wrong/outdated, because once you look into it, you will notice that companies adapt / change their model regularly. If you need to understand one specific company’s model you need to dig into it individually at that time.

There are three basic open source licenses to be distinguished: permissive, weak copyleft and copyleft.

A quick high-level note on the major license effects

Copyleft – major point is that derived works must be open sourced with a compatible copyleft license, meaning any advancements and changes to the work will be contributed back to the community and freely available for unrestricted use.

Weak Copyleft – the weaker copyleft refers to licenses where not all derived works inherit the just described copyleft effect; typically used in software libraries, e.g. a database library used in app development, so the library can be used in a mobile app without needing to contribute the whole app to open source; only changes to the database library itself would carry the copyleft effect.

Permissive – a permissive open source license allows you to do anything with the source code including keeping derived works to yourself and commercialising on it.

	Description	Examples	Note
Paid Support	Providing paid support, trainings, certificates	RedHat	Where has this approach been working – as a pure paid support approach – ever since Red Hat?
Open Core	The core product is free and open source, extra features are paid; have an open-source core and sell closed-source features on top of it	SugarCRM, MySQL	It is basically the widely successful freemium model just with open source; typically you expect the large majority of users to use it for free. The open source part of course enables anyone to build the same features as you
Dual Licencing	The free open source sw uses a copyleft license, whereas the paid license is a commercial license without copyleft effects	MySQL, elastic	This kind of license enables you to monetize your commercial (typically bigger users) and still enables the community to expand the product landscape and innovate based on the code base
Delayed Open Source	All code will be fully open sourced with a time delay (details and timings vary)	MariaDB, Cockroach DB	The effect depends also on the licenses used, but typically it protects you from competition for a given time frame, so only you can exploit your development commercially and gain market share / develop an advantage based on market entry time. At the same time it reduces the risk for adopters, because they know the code will become available to them
Open SaaS	Offering the software open source and hosted as a service (SaaS), which is the primary source of revenue allowing anyone to do the same with the software with a permissive license (self-host or host for others)	WordPress, Sharetribe, MySQL, MariaDB	This model has been the major point of discussion in the last 3 years and is seen by many as the holy grail for monetizing open source software; it also triggered many companies to move away from an open source licensing model as large cloud providers can easily host an open source product at better rates
“Closed SaaS”	Strictly speaking / officially not “open source”. Offering the solution open source and hosting it as a service (SaaS) while NOT allowing anyone to host it, often times unless they contribute the whole solution back to open source (copyleft effect))	MongoDB, elastic, Cockroach DB	The first license that built this specific copyleft-effect into its license was MongoDB (SPSL). The license has since been adopted by e.g. elastic, …. Since then similar licenses have been developed. OSI did not approve the license as an official open source license.
“Ad model”	For lack of a better name, I called it “Ad model”; it’s really having so much reach and traction that companies pay for customer access through your solution or similar co-operations	AdBlock Plus, Firefox	Can take many variations: For instance, the open-source application AdBlock Plus gets paid by Google for letting whitelisted acceptable Ads bypass the browser ad remover. Or, in 2014 Yahoo struck a deal with the Mozilla Corporation to make Yahoo the default search engine in Firefox

A look at the open source market

Name	Founding Year	Funding Summary	Started with Open Source (license)	Open Source Evolvement	Devtool	Open to contributions / CLA	HQ*	Notes / Story synopsis
MongoDB	2007	6 funding rounds with a total of $311M IPO was in autumn 2017; valuation $1.6B	started with AGPL	Created SSPL in 2018 causing much debate in the community. SSPL is not an open source license	Database	“we own 100% of the IP”; 99.9% developed in house and the few contributions accepted were from people who signed a CLA	US-based	According to statements fromMongoDB, adoption went up after the license change (15 mill dwlds, more than in the prior 10 years together). In 2016 they launched their database-as-a-service offering, which is considered the game changer w. regards to building a business. Until Oct 2017 MongoDb downloads were >30M with 10M from the prior 21 months.
Data Bricks	2013	Total funding 1.9B; last round: Series G; Feb 2021 $1B	proprietary PaaS	their main service is proprietary, but they use a lot of open source software and have a strong footprint in the open source community	Backend	NA	US-based	“Databricks is the original creator of some of the world’s most popular Open Source data technologies” – open source is a large part of their positioning and marketing. However, it seems their main offering, while based on open source, is proprietary. So, not an open source business as defined here.
elastic	predecessor released in 2004; first elasticsearch released in 2010; incorporation only in 2012	Total funding $162M; last round was a series D; elastic did IPO in autumn 2018	started with Apache 2 for for elastic search (which was the original main product)	Last license change in 2021: You can now choose between the proprietary elastic license or SSPL; so stritly spaking not open source anymore	Devtool	CLA	US-based	2018: elastic IPO –> shares doubled the first day. Note: With so many different products (not a single product company), the open source strategy is harder to grasp.
Confluent	2011	Total Funding Amount $455.9M, last round: series E	Unlike Apache Kafka which is available under the Apache 2.0 license, the Confluent Community License is not open source and has a few restrictions	Kafka is open source, Confluent isn’t	Devtool	NA	US-based	“Founded by the team that originally created Apache Kafka” – the team behind Confluent contributed a lot to open source prior to Confluent, but the Confluent code itself isn’t open source as far as we understand. They heavily rely on other open source software for their tech stack though.
RealmDB	2011, before the founders did “TightDB” on which the Realm DB was based	4 investment rounds. Then MongoDB acquired them for $39M on Apr 24, 2019	started out closed; then open sourced the database and went for the open core model, then subsequently open sourced the Sync solution too, going for the hosted (SaaS) model	from closed to open core to open SaaS; acquired by Mongo to push their backend offerings and complement with an edge and sync (serving Mobile and IoT better)	Database	looks like they accepted contributions	Started in Europe, but HQ went to the US when joining YC 2014; it was since bought bei MongoDB	The founders both left the company the year before it was acquired by MongoDB. The acquisition prize was a little less than what Realm had raised in the years before. The Sync solution is now tied to using the Mongo servers / cloud and a huge part of their push for the IoT market.
SQLite	2000	Bootstrapped	Public Domain, which we always considered one of the most “open source” ways to open source but in the light of recent discussions around the SSPL license, strictly speaking it is at least not OSI-approved	Public Domain, mainly monetize big corporates for being in a Consortium; also offers services and since xxxx? encryption (basically paid feature); our guess is that this is not really a repeatable business model	Database	Richard Hipp owns all IP, 99% is developed by himself; very limited outside support (2 part-time freelancers that we are aware of, both don’t have any rights to the IP)	US-based (privately held by Hipp, Wyrick & Company, Inc (author: Richard Hipp and all stock held by his wife G. Wyrick; both work for the company)), HQ	The company has always been and still is run by Richard Hipp and his wife; from a development perspective it is a one-man-show. Richard wrote SQLite himself, as far as we are aware they have no other employees apart from 2-3 part-time supporters for specific versions; very special Open Source Story.
Couchbase Lite	2009 – Couchbase, Inc. is a merger of Membase + CouchOne in 02.2011; both former companies were started 2009 and had funding	251 million USD total funding; 8 rounds with latest Series G for $105 million	Apache 2	Delayed Open Source	Database		US-based (both entities were US-based already before the merger)	Couchbase now mainly sells Couchbase Servers; Couchbase Lite is the smallest part of their business; in 2020 there seemed to be a shift towards the Sync Gateway and Edge Computing market in communication; however, the main business still seems be on the server side and based on cloud lock-in.
redis	2009	Total Funding Amount $246.6M	redis the database itself is and always was BSD; redislabs is the company that has secured certain rights for redis and sells extensions and add-ons under several licenses, they changed from APGL to Apache 2.0 with Common Clause to a proprietary license called “Redis Source Available License”	redis itself is BSD but features / extensions around it from RedisLabs are licensed uner prorietary licenses	Database	Any contribution needs a CLA that is provided by redislabs; we believe anything committed under this CLA could also be used in redislabs proprietary products (which typically is the same for anything committed under a permissive license, but which has attracted some criticism from the OSS community)	Redislabs is US-based. Salvatore Sanfillipo (antirez) was always bsaed in Europe; redislabs originated in Israel	RedisLabs is the commercial entity that markets redis; redis was largely developed by Salvatore Sanfilippo. He left redis as a maintainer in 2020.
RedHat	1993	bought by IBM in 2019 for $34 billion; before that they had raised $240.7M	Linux, which was the core of the success of RedHat, is GPL (though of course not the company’s decision)	RedHat is a huge company, definetely not a single product company, and thus also does not really fit into this matrix, however, it is THE example for successful commercialisation of open source and we feel the matrix would lack without it	Backend / Data centric	we believe you can contribute to most (all?) projects without a CLA	US-based	Read here why there will never be another Red Hat (and there is no “Red Hat Model”). Note that of course the Red Hat founders did not write Linux (on which the majority of their success is based), but at the very least they (as well as VA Linux) gave option shares to Linus Torvald out of gratitude (at lest not out of obligation). When both companies successfully IPOd, Linus made 20 Mill USD (in total) from both sales.
MySQL	1995 (development started already in 1994)	Total Funding Amount $39.8M, sold to Sun in 2008 for 1 USD billion	started out with AGPL; several license adaptions and changes in the open source business model over the years, e.g. for a long time they had a 2 year delay for the open source version, but changed that to no delay at some point.	Dual Licensing and Paid Support	Database	Yes, even though called OCA (Oracle Contributor Agreement)	Sweedish company until it was acquired by Sun Microsystems in 2008 (who then were acquired by Oracle)	The founders forked the latest MySQL version when Oracle acquired it. Most of the original database code base was developed by Michael Widenius; with regards to database technologies a pattern emerges: Often the core / most of the base technology is developed by one person – as building a database is a rather huge endeavor that’s kind of striking, isn’t it? BTW: MySQL is named after Monty Widenius daughter (“My”)
Hyper	2010 (academic research project at TUM)	undisclosed	proprietary, not open source	None	Database	NA	EU-based; German “university spinoff” acquired by Tableau very early	2016: HyPer acquired by Tableau. Terms of the deal were undisclosed
ParStream	2011	acquired by CISCO in November 3, 2015	proprietary, not open source	NA	Database	NA	Originally EU-based (German), then moved to US in 2012, acquired by Cisco in 2015	Cisco ParStream is no longer offered as a stand-alone product. The functionality of Cisco ParStream is now part of Cisco Kinetic.
Cockroach DB	2015	Series E in Jan 2021 for $160M	Apache 2.0, plus a proprietary license for enterprise features	Started as open core, now a form of closed SaaS with delayed open source: They changed to a proprietary license in 2019, called BSL, which prohibits users from offering CockroachDB as a service (DBaaS, SaaS), and each release converts to an open source license after three years. CockroachDB is therefore officially not considered open sorce anymore	Database	CockroachDB received significant contributions from the community (“we have had over 1590 commits from over 320 external contributors across all our open source repositories” (2020)), CLA: Yes	US-based	In June 2019, Cockroach Labs announced that CockroachDB would change its license from the free software license Apache License 2.0 to their own proprietary license, known as the Business Source License (BSL), which forbids “offer[ing] a commercial version of CockroachDB as a service without buying a license”, while remaining free for community use.
Berkeley DB	1994	Acquired by Oracle in 2006	BSD and Sleepycat Public License (a permissive OSS license)	Oracle changed to dual licensing with APGL and a commercial license	Database	NA	US-based	It is still used in many routers and gutfeel is that the market share in that specific area is good. Unfortunately, no numbers available.
GitHub	2008	In 2018 Microsoft bought GITHUB for $7.5 billion.	proprietary, not open source	NA	Backend / Data centric	NA	US-based	Microsoft bought GitHub for the developer access; that would not have changed if it would have been open source and I do wonder what would have happened to GitHub if it would have been open source; one thing is for sure: GitLab wouldn’t have been able to position themselves as the open source alternative; however: the closed source model worked for them well, even though it is a developer tool.
GitLab	development started in 2011; incorporated only in 2014	$434.2M Series E	completely open source (MIT license)	Now: Open Core Model; Community Edition: MIT License Enterprise Edition: Source-available proprietary software	Backend / Data centric	Originally CLA, now dropped and instead the code must be committed under the same license as the feature is (mainly Apache 2.0) plus a DCO	US-based (development was started in Europe, the founders incorporated in the US in 2014 when joining YC)	GitLab used being open source as a strong positioning factor against GitHub (which was never open source). It was an odyssey to find a sustainable business model (and it seems it is not SaaS). Note: The pure service model and the donation model did not work for them. Again: The code base of the core system was by and large developed by one person.
MariaDB	2009	Total Funding Amount $123.2M	Dual licenscing with GPL license, version 2 and a prorietary source available license for some parts	They evolved their dual licensing approach to using the proprietary source avaiable license (BSL)	Database	Yes, and the CLA is shared under a creative commons license that allows you to use it as you like https://mariadb.com/kb/en/mca/	Sweedish company	10 years after it was forked, MariaDB has 20M users, a fast growing database business and has >€100m backing. Note: The pure service model as well as the donation model did not work for them.

Building an Open Source business Exec Summary – TL; DR

There is a lot of evidence that open source companies struggle with open source models and licenses – this is also true for successful companies
There is no “Red Hat Model” – just selling services has rarely worked
The donation model typically hasn’t worked for open source companies, e.g. GitLab and MariaDB, so it is not astonishing that GitHub sponsorships don’t work out great for most maintainers. Also note: GitHub sponsorships may put you in a bad legal position depending on where you are based
There is a trend from successful open source companies towards Source Available licenses instead of “official Open Source licenses”, e.g. MongoDB, elastic, CockroachDB, …
There is an indication that successful open source companies are US-based (even if founded / started in Europe), which we believe is due to the funding opportunities provided in the US: 1) the US provides generally more funding (more and bigger funding opportunities; there is lots of market research on that), 2) US VCs and Silicon Valley have the reputation to also fund at earlier stages, e.g. idea stage, and companies with traction (instead of revenue), investing in a longterm perspective. Traditionally, European investors don’t.
Public domain is strictly speaking also not considered to be an open source license 😮 (at least not if it needs OSI-approval; does it? 🤔)
While Open and Closed SaaS seem at this moment to have been the most successful models, it is no holy grail and definetely does not work for everyone, e.g. it didn’t work as the sole business model for GitLab

Conclusion

The open source market lacks flexibility and transparency from a licencing / legal perspective, and ever more Source Available licenses don’t help: A “license stack” with building blocks like the Creative Commons would be helpful to mark software easily and clearly with regards to the main terms, e.g. “source available”, “free for commercial use”, “attribution necessary” etc. It would help maintainers and users alike, but needs bigger entities to drive this (like an OSI).

The open source market also needs more balance, at the very least more understanding and “love” towards maintainers. More finanical support as well as other ways of giving back to demonstrate the appreciation of well-maintained repos and great free software, will keep the ecosystem healthy and thriving. That’s a community effort; everyone can contribute.

Sources

1. https://openforumeurope.org/event/accelerating-the-eu-economy/
2. https://db-engines.com/en/ranking_osvsc
3. https://www.synopsys.com/blogs/software-security/open-source-trends-ossra-report/
4. https://agcpartners.com/wp-content/uploads/2021/03/AGC-Open-Source-Mar-2021.pdf
5. https://agcpartners.com/wp-content/uploads/2021/03/AGC-Open-Source-Mar-2021.pdf
6.https://www.bvp.com/atlas/roadmap-open-source
7. https://www.bvp.com/atlas/roadmap-open-source
8. Index Open Source Summit (2021)

« Older Entries

Next Entries »