Generative AI, Databases, and the Multidimensional Space.

Why is Everyone Talking About Graph and Vector Databases These Days?

In today's dynamic realm of AI and data management, chatter about graph and vector databases is gaining momentum. Take Tanzu Hub as a great example — it's driven by a cloud-scale, graph-based database. Tanzu Gemfire, a powerful application data cache, recently unveiled a Vector Database Extension. This enhancement facilitates the storage, indexing, and querying of vector embeddings tailored for AI applications. As for Tanzu Greenplum, this data warehousing, analytics, and AI platform is now integrating an Automated Machine Learning Agent.

As I prepare to present at #ExploreBarcelona next week, I sometimes wonder if I've truly grasped the depth of these technologies. To demystify them, I sought guidance from my trusted #GenAI allies, #chatGPT and #GoogleBard, and embarked on an enlightening journey.

Graph and vector databases are two distinct types of databases that can be used to support AI (Artificial Intelligence) workloads in different ways.

Why do we need Graph Databases?

#GraphDatabases are designed to store and manage data in a graph structure, consisting of nodes, edges, and properties.

• #Nodes: Nodes represent entities or objects in the dataset, often depicted as circles or points, and store information about these entities.
• #Edges: Edges represent relationships or connections between nodes, often depicted as lines or arrows, and describe how nodes are related to each other.
• #Properties: Properties are key-value pairs associated with nodes or edges, providing additional details or attributes about the entities and relationships in the graph.

For example, in natural language processing (NLP), understanding context and relationships between entities is crucial. Hence, there is a need for graph databases to create and query knowledge graphs.

Graph databases are great at modeling and querying complex relationships and connections within data. Therefore, they underpin many social media networks like Twitter and recommendation engines like Amazon, Netflix, Spotify, and fraud detection systems like CyberSource, Forter and Splunk

• Recommendation systems use Graph Databases to model user-item interactions. Nodes represent users and items, and edges represent interactions (e.g., watched a movie, bought a product). By analyzing the graph, recommendations can be generated based on user behavior and similar user profiles.
• Social networks extensively use graph databases to uncover patterns, influencers, and communities within the social networks. This is beneficial for tasks like sentiment analysis, viral content prediction, and identifying influencers.
• Fraud detection flags unusual patterns and relationships among transactions and accounts.

Why do we need Vector Databases?

#VectorDatabases, on the other hand, are designed for storing and querying high-dimensional vector data efficiently. These databases are particularly useful when dealing with data representations like #embeddings, commonly used in AI workloads.

Vector databases excel at similarity search by efficiently calculating the #similarity between vectors, making them ideal for tasks like image similarity, content recommendation, and even searching for similar documents.

Many AI models use vector embeddings to represent data, such as word embeddings in NLP or image embeddings in computer vision. Vector databases provide a way to store and retrieve these embeddings, allowing for fast and scalable access.

Vector databases can also handle time series data efficiently. This is crucial for AI applications like predictive maintenance, anomaly detection, and forecasting, where historical data is used to make predictions.

Graph databases + Vector databases = better together

Now if we combine graph and vector databases we are able to create a conversational experience that offers richer, more context-aware, and personalized responses to user queries than previously thought possible. Graph databases, with their intricate web of nodes, edges, and properties, prove indispensable for modeling complex relationships, as exemplified in natural language processing and recommendation systems. They lie at the core of social networks, enabling sentiment analysis, viral content prediction, and fraud detection.

On the other hand, vector databases excel in efficiently handling high-dimensional vector data, a cornerstone of AI applications. They unlock the potential of similarity search, propelling image similarity, content recommendation, and document retrieval to new heights. Vector databases also prove their mettle in managing time series data, vital for predictive maintenance, anomaly detection, and forecasting.

Yet, it's the synergy between graph and vector databases that holds the promise of transforming user experiences. By bridging structured knowledge, semantic searches, and adaptive interactions, this integration augments AI's utility across various domains. 

Okay, we've talked quite a bit about vectors, but what are they? How do they look?

A vector is a one-dimensional array of numbers. It represents data along a single dimension or axis. For example, a simple vector could be [1, 2, 3], representing data along a single dimension, such as three values in a sequence.

But here, we're talking about high-dimensional vectors. In 2D, vectors represent a point on a graph:

 In 3D, they represent a point in space:

Vectors in AI, especially those used for embeddings, represent data points in a space with a large number of dimensions, termed as high-dimensional space. Unlike easily visualized 2D or 3D points, these vectors in high-dimensional spaces are challenging to picture. Instead, they can be thought of as a collection of numerical values, typically organized as lists or arrays. Each element of the vector corresponds to a specific dimension in the high-dimensional space. 

In natural language processing (NLP), a high-dimensional vector could represent a word or phrase using word embeddings, where each dimension captures semantic relationships between words. In computer vision, a high-dimensional vector might represent an image using features extracted from different regions of the image, such as pixel values, color histograms, or deep learning embeddings.

Watch this short video to get your head around it:

https://www.youtube.com/watch?time_continue=191&v=wvsE8jm1GzE&embeds_referring_euri=https%3A%2F%2Fars.electronica.art%2Fcenter%2Fen%2Fvisualizing-high-dimensional-space%2F&feature=emb_logo

In summary, I hope this blog post has helped shed some light on the intriguing realm of graph and vector databases, unraveling their vital roles in AI workloads, and giving you a glimpse into the abstract yet powerful world of high-dimensional vectors. As technology continues to advance, these topics will undoubtedly remain at the forefront of discussions, shaping the future of AI and data management.