The System Of Intelligence

Introduction

With the advent of big data and artificial intelligence, the world is witnessing a paradigm shift in how organizations process and utilize data to make informed decisions. The traditional systems of record, such as CRM and ERP, have evolved into systems of engagement, which empower businesses to connect with their customers and employees seamlessly. Later, systems of business intelligence emerged to provide reporting tools, enabling users to extract valuable insights from data. However, the next generation of data analytics solutions, known as the system of intelligence, is poised to revolutionize the way organizations interact with data.

In this article we will go through the evolution of data systems, and how in the future data would be consumed by businesses. Also, I will show a small demo on it.

The Evolution of Data Systems

Data systems have come a long way in the past few decades. Traditional systems of record, such as CRM and ERP, were designed to capture and store data. These systems were useful for basic data management, but they lacked the analytical capabilities required to extract insights from the data.

This limitation gave rise to the systems of engagement. These systems were designed to improve customer engagement by providing seamless communication channels between businesses and their customers, allowing businesses to gather valuable customer feedback and insights.

However, despite their ability to improve customer engagement, systems of engagement were not capable of delivering in-depth analysis of the data. This led to the emergence of systems of business intelligence. These systems were designed to extract insights from the data by providing powerful reporting and analytics tools. Here is where all the data cleaning, transforming data, data warehousing, data pipelines, and the whole bandwagon of the data ecosystem come in.

Systems of business intelligence enabled businesses to make data-driven decisions by visualizing and analyzing the data. These tools allowed users to generate reports, dashboards, visualizations, etc that provided insights into the data.

While systems of business intelligence are useful, they had limitations. These systems still required a significant amount of human intervention to generate insights. They were often complex and required specialized skills to operate.

This led to the emergence of the system of intelligence. The system of intelligence is designed to leverage advanced technologies.

Introduction to System of Intelligence

The system of intelligence (SOI) is a next-generation data analytics solution that is designed to extract relevant insights from data by leveraging advanced technologies. At the heart of the system of intelligence is the use of language and learning models (LLMs) that enable computers to understand natural language and learn from the data. They are capable of processing vast amounts of unstructured data and extracting valuable insights from it.

The system of intelligence is designed to work in a way that is different from traditional data systems. Rather than requiring users to know how to extract insights from the data, the system of intelligence is designed to ask questions and provide insights in a way that is easy for non-technical users to understand.

To achieve this, the system of intelligence is built on a vectorized search architecture. This architecture enables users to ask questions in natural language and receive relevant insights in real time. The system of intelligence is designed to learn from the data and continually refine its search algorithms to provide more accurate insights over time.

The benefit of the system of intelligence is its ability to provide context to the data. By using LLMs, the system of intelligence is capable of understanding the nuances of language and providing insights that are relevant to the context of the question being asked. This capability enables businesses to make more informed decisions and take actions that are aligned with their goals and objectives.

Let's implement it!!

We would be doing a small proof of concept using below tools and packages

• LlamaIndex : This package augments LLMs with our own private data.
• OpenAI: Using OpenAI API to embed the documents.
• Gradio: To demo this POC with a web interface

import os
os.environ["OPENAI_API_KEY"] = "<your-api-key>"

import gradio as gr
from gpt_index import 
    SimpleDirectoryReader,
    GPTSimpleVectorIndex,
    LLMPredictor,
    PromptHelper,
)
from langchain.chat_models import ChatOpenAI
        

def build_index(directory_path):
    """
    This functions read the documents like PDF, Text Files, etc
    from the directory_path and generates embeddings using OpenAI 
    GPT 3.5 Turbo Model
    The embeddings of the documents are stored in index.json file
    """
    prompt_helper = PromptHelper(
        max_input_size=4096,
        num_output=512,
        max_chunk_overlap=20,
        chunk_size_limit=600,
    )


    llm_predictor = LLMPredictor(
        llm=ChatOpenAI(
            temperature=0.7, model_name="gpt-3.5-turbo", max_tokens=512
        )
    )


    documents = SimpleDirectoryReader(directory_path).load_data()


    index = GPTSimpleVectorIndex(
        documents, llm_predictor=llm_predictor, prompt_helper=prompt_helper
    )


    index.save_to_disk("index.json")


    return index
        

def chatbot(input_text):
    """
    This the implementation of chat bot, for the given input_text
    it queries the index generated. We can use vector DB like
    ChromaDB, PineCone, Weaviate. Here, we are using a JSON file instead.
    """
    index = GPTSimpleVectorIndex.load_from_disk("index.json")
    response = index.query(input_text, response_mode="compact")
    return response.response
        

# Creating the web interface
interface = gr.Interface
    fn=chatbot,
    inputs=gr.components.Textbox(lines=7, label="Enter your Query"),
    outputs="text",
    title="Custom trained Chatbot",
)
index = build_index("docs")
interface.launch(share=True)(        

Let's ask ChatGPT about DataSlush

Let's Augment our Chatbot which uses OpenAI Model to understand the query and answer accordingly by providing the document about the company.

About DataSlush:
We are Data and Analytics Consultancy focused on leadership, process improvement in
data landscape, implementation, and advanced analytics which assist organizations in
successfully complete their data initiatives. We enable organizations on managing,
comprehending, and acting on their data.
We believe that having complete transparency, keeping all stake holders in loop, and
understanding business pain points always help in delivering the project with less friction
We emphasize and ensure the deliver of project in such a way that empowers the clients
to self manage the project.Our Clients love that we own the project and it's responsibilities and solves complex
probelms with commitment and competency.




Company Vision:
To enable organisations on managing, comprehending, and acting on their data.


Company Mission:
With complete transparency solving complex probelms with commitment and competency.


Company Service:
Data Analytics, Data Visualization & BI
Data Engineering
Data Science
Solution Engineering
MLOps
Cloud Architecting
Tag Management & Analytics Implementation        

Here is the Demo

Now imagine, you as a business or data owner asking questions like,

Hey, calculate the last quarter's ROI on marketing campaigns, and consider only Google Ads campaigns that lead to Website Signups.

Give me the segment of customers, who have abandoned cart in last 4 hours, and chose one of the email templates made by John Doe to send them from the marketing repository.

I don't know how such things would be possible right now but are probably to happen. Can't wait to do more experiments with open-source projects like Dolly 2.0, StableLM, LangChain, and Vector DBs.