Home Tech blog Talking with OpenAI GPT about J-labs with the help of LangChain

Talking with OpenAI GPT about J-labs with the help of LangChain

Marcin Tarka

11.06.2024

ChatGPT is a super useful tool running on top of powerful GPT-4 and GPT-3.5 models. It can assist your writing, support you with a huge base of knowledge and of course assist in codding. But still it has a significant flaw – it has no access to current knowledge from the open internet. It includes information up until September 2021 and was not trained on full human knowledge up to this date.

So what if you would like to create your own chat-bot based on the GPT model that would have access to an actual source of knowledge? Or maybe to some niche that wasn’t even used during GPT training? Or your company’s internal knowledge base?

For those use-cases LangChain library comes to rescue. In this article I will show you how to build a basic chat-like script that will maintain a dialogue about j‑labs.

LangChain basics

So what exactly is LangChain? As its official website says:

LangChain is a framework for developing applications powered by language models. We believe that the most powerful and differentiated applications will not only call out to a language model, but will also be: 1. Data-aware: connect a language model to other sources of data 2. Agentic: allow a language model to interact with its environment

In this article I will not use LangChain agents, but making Large Language Models (LLM) data-aware is exactly what we are looking for.

LangChain’s process of providing knowledge to LLMs is both interesting and simple. The process is following: 1. LangChain connects to source of external knowledge. This can be for example a list of text files 2. LangChain splits external knowledge into chunks and creates embeddings out of it. If you don’t know what embedding is I recommend you an article from TensorFlow guide. At this stage user can choose embedding model. 3. After calculation LangChain stores all embeddings in a vector database. Some popular open source engines can be used like Chroma or Faiss. This point ends the phase of data preparation 4. When LangChain is prompted with some user input that it looks for corresponding knowledge in vector database and constructs a final prompt to LLM using a template. Below you can find one of the templates implemented in LangChain. It will be used in our code:

Use the following pieces of context to answer the question at the end. If you don't know the answer, just say that you don't know, don't try to make up an answer.

{context}

Question: {question}
Helpful Answer:

Looks very promising, isn’t it? As you will quickly find in fact it does the job, but keep in mind that this approach has some limitations. The mayor one is that it passes all data in one prompt. Prompt size is limited and in moment of writing this article it was 4096 tokens. LangChain has some features that try to minimize this disadvantage (like map_reduce, map_rerank, and refine chain types) but it is not covered in this article.

Prerequisites

There are a number of prerequisites that you will need to follow this tutorial. The most important one is an OpenAI API key. You can get one by registering on OpenAI page. After logging-in go to sub-page https://platform.openai.com/ and click on the upper right user icon, then choose ‘View API keys’. You should be redirected to a page where it is possible to view and generate API keys. Another prerequisite is Poetry. This is a very simple in use packaging and dependency management system for Python. Of course it is possible to compile presented code just by using plain Python installation but I highly recommend using Poetry.

Creating working environment

Lets create a new folder containing Poetry configuration by typing in the terminal: poetry new jlabs-chat. Now its time to define project dependencies, so lets open pyproject.toml file and add following ones:

[tool.poetry.dependencies]
python = "^3.10"
openai = "^0.27.7"
langchain = "^0.0.189"
transformers = "^4.29.2"
chromadb = "^0.3.26"
tiktoken = "^0.4.0"

As I mentioned before, if you are not planning to use Poetry then you can just install those libraries using pip. If you use Poetry it is now enough to call a command

poetry install

in projects folder, and then to run any Python script just call

poetry run python your_script_name.py

Knowledge base

Now its time to gather knowledge for our chat. For a sake of simplicity I will just use a single txt file. I copied all textual content from https://www.j-labs.pl/en/ and pasted it to my text file which I called jlabs_en.txt and stored in project folder. Exemplary part of the file looks like that:

Who we are?
We are reliable. We devote ourselves to what we do and we maintain high quality.

We go with merit approach. We build a team of experts in specific fields, who continue to develop.

We focus on technology. In the Java “laboratory” (hence the name j‑labs!), equipment and results go hand in hand with technology development.

Knowledge worth sharing. J-labs brings together a committed community of people who are united by their passion for technology. Talk4Devs is a festival of knowledge from engineers for engineers.
We have been meeting since 2014.

Basic script

Lets create a simple python script that will serve the simplest possible knowledge supported chat using LangChain. The code you can find below:

from langchain.embeddings.openai import OpenAIEmbeddings
from langchain.vectorstores import Chroma
from langchain.text_splitter import CharacterTextSplitter
from langchain.llms import OpenAI
from langchain.chains import ConversationalRetrievalChain
from langchain.memory import ConversationBufferMemory
from langchain.document_loaders import TextLoader

openai_api_key = '<your key goes here>'

loader = TextLoader("jlabs_en.txt")
documents = loader.load()

text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=100)
documents = text_splitter.split_documents(documents)

llm = OpenAI(temperature=0, openai_api_key = openai_api_key)
embeddings = OpenAIEmbeddings(openai_api_key = openai_api_key)
vectorstore = Chroma.from_documents(documents, embeddings)
memory = ConversationBufferMemory(memory_key="chat_history", return_messages=True)

qa = ConversationalRetrievalChain.from_llm(
    llm,
    vectorstore.as_retriever(),
    memory=memory,
    )

try:
    while True:
        query = input("> ")
        result = qa({"question": query})
        print('J-chat:', result['answer'])
except KeyboardInterrupt:
    print('\nBye!')

Lets go through it step by step.

Fist of all I declared a variable with the API key. I described in one of the previous sections how to get one.

Next loading of our knowledge document is performed. It is done using TextLoader which is a simple class for loading from text files.

CharacterTextSplitter class is used to convert a text of arbitrary length to a series of documents. Text splitter cuts our input text file into chunks containing 1000 character. Each chunk overlaps on a previous one by 100 characters. This step is needed because of limitations of prompt size which I mentioned before. Our script will work around this problem by dividing text into chunks and then feeding only the best matching parts to prompt.

After that it is time to declare key objects of the script, llm of class OpenAI which represents OpenAI GPT LLM model, embeddings of class OpenAIEmbeddings which represent embeddings model from OpenAI.

In the next step embeddings and documents produced by splitter are used to create a vectorstore object which is our vector database. To keep things simple embeddable Chroma database is used but LangChain supports a vast list of databases.

Next memory buffer for conversation is declared. It keeps context of a given conversation. Please note that this code is not ready to be used in multithread context because always uses a hardcoded memory key.

Now it is a time to declare the most important object in LangChain library which is a chain. As a definition of chain from official LangChain webpage says it is a structured assembly of components for accomplishing specific higher-level tasks. Our chain is a ConversationalRetrievalChain which is developed for conversations with humans, has an ability to retrieve knowledge from vector database and has a context memory.

There are many other chains in LangChain lang math chain which performs mathematical tasks expressed in natural language, text moderation chain, chain which retrieves data from SQL database or tagging chain which can tag documents.

At the bottom of the script, to make it practical, I declared just a simple loop that waits for an input and prints answer on a terminal. It can be finished by keyboard interrupt (ctrl+c).

Testing and tweaking the answers

So, lets have a short conversation with our j‑chat.

> Describe organizational culture of j‑labs.
J-chat: The organizational culture of j‑labs is base on values such as reliability, merit approach and technology. It is an environment where work gives satisfaction and personal growth, and collaboration is initiated with people who share the same beliefs. J-labs also brings together a committed community of people who are united by their passion for technology.
> Is j‑labs a good place to work?
J-chat: Working at j‑labs is a rewarding experience. The company is created by engineers for engineers and values reliability, merit approach and technology. They invest in people and create an environment where work gives satisfaction and personal growth. They bring together a committed community of people who are united by their passion for technology. The offer specialist outsourcing, IT staff augmentation and IT team augmentation.
> Cool so you would recommend me applying?
J-chat: Yes, j‑labs is a great company for engineers to work with. The have a committed community of people who are passionate about technology. The focus on IT engineers with narrow, specific specializations and several years of international experience. They also have a merit approach and focus on technology. THe offer IT Team Augmentation, where ou can collaborate with their specialists to deliver comprehensive solutions.
> How can I apply then?
J-chat: Yoy can contact j‑labs at j‑labs@j-labs.pl to apply.
> How many employees j‑labs has?
J-chat: I don’t know.

On one hand it is quite impressive. It generates nice conversation based on texts from j‑labs webpage. On the other hand communication is quite cold. If we would imagine an IT company having a chat on its webpage then probably this company prefers the chat to form messages as it were an employee. The form in which our chat admits that it doesn’t know the answer is also quite problematic, it is not very polite nor informative.

What causes this behavior? Well, lets take a look at conversation template which I presented earlier. It does not specify that chat should impersonate itself as an employee. It also enforces chat to give very short answers to questions where no direct knowledge is provided.

So lets customize the template:

from langchain import PromptTemplate
from langchain.embeddings.openai import OpenAIEmbeddings
from langchain.vectorstores import Chroma
from langchain.text_splitter import CharacterTextSplitter
from langchain.llms import OpenAI
from langchain.chains import ConversationalRetrievalChain
from langchain.memory import ConversationBufferMemory
from langchain.document_loaders import TextLoader

openai_api_key = '<your key goes here>'

loader = TextLoader("jlabs_en.txt")
documents = loader.load()

text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=100)
documents = text_splitter.split_documents(documents)

llm = OpenAI(temperature=0, openai_api_key = openai_api_key)
embeddings = OpenAIEmbeddings(openai_api_key = openai_api_key)
vectorstore = Chroma.from_documents(documents, embeddings)
memory = ConversationBufferMemory(memory_key="chat_history", return_messages=True)

template = """
Act as if you were a j‑labs employee, so when writing about j‑labs don't use form 'they' but 'we'.
Give warm and enthusiastic answers.
Use the following pieces of context to answer the question at the end.
If you don't know the answer, just say that you don't know, don't try to make up an answer and suggest contacting j‑labs.

{context}

Question: {question}
Warm and enthusiastic answer written by j‑labs employee:
"""

qa = ConversationalRetrievalChain.from_llm(
    llm,
    vectorstore.as_retriever(),
    memory=memory,
    combine_docs_chain_kwargs={
        "prompt": PromptTemplate(
            template=template,
            input_variables=["question", "context"],
        ),
    }
    )

try:
    while True:
        query = input("> ")
        result = qa({"question": query})
        print('J-chat:', result['answer'])
except KeyboardInterrupt:
    print('\nBye!')

In the presented snippet I passed a custom template to the chain. It specifies that chat has to perform conversation of it were an employee. It also looses a bit expectations about situation where chat doesn’t know the answer. Lets run whole dialog once again:

> Describe organizational culture of j‑labs.
J-chat: At j‑labs, we have created an environment where work gives us satisfaction and personal growth. Our organizational culture is based on values such as reliability, merit approach and technology. We strive to bring partners closer to their business goals and continually improve the expertise of our j‑specialists. We invest in people, because the are at the heart of j‑labs. We also great a committed community of people who are united by their passion for technology. Talks4Devs is a festival of knowledge from engineers for engineers that we have been organizing since 2014.
> Is j‑labs a good place to work?
J-chat: At j‑labs, we create an environment where work gives us satisfaction and personal growth. We are reliable and devoted to what we do, and we maintain high quality. We focus on technology and invest in people, because they are at heart of j‑labs. We go with a merit approach and build a team of experts in specific fields, who continue to develop, We also bring together a committed community of people who are united by their passion for technology. We have been meeting since 2014 and we are proud to have joined the largest companies in the world that have completed their digital transformation together with us. Al of this makes j‑labs a great place to work!
> Cool so you would recommend me applying?
J-chat: Absolutely! We are always looking for talented and passionate engineers to join our team. We offer a great working environment, where you can develop your skills and work on interesting projects. We also provide a competitive salary and benefits package. So if you are looking for a great opportunity to work with a committed community of people, then j‑labs is the place for you!
> How can I apply then?
J-chat: At j‑labs we are always looking for talented and passionate engineers to join our team. if you are interested in applying, please send us your CV and a cover letter to j‑labs@j-labs.pl. We look forward to hearing from you and learning more about your experience and skills.
> How many employees j‑labs has?
J-chat: At j‑labs, we are proud to have a team of experienced engineers and specialists who are passionate about technology and committed to delivering the best possible product to our partners. We currently have over 100 employees, and we are always looking for new talent to join our team. If you would like to learn more about our team and our services, please don’t hesitate to contact us at j‑labs@j-labs.pl

As you can see now it is much better aligned to our use-case. Now chat impersonated itself as if were an employee of l-labs. Answers are longer and more enthusiastic.

There is also a side effect that in some cases may be considered as a negative one. While answering the last question the chat came up with an answer which is not directly deducible from the knowledge base. This issue can be mitigated by providing knowledge which directly refers to a number of l-labs employees.

Please remember that full code of this app you can find on my github account.

What to consider building real-world solution

Please keep in mind that it was a very simplified example which operates on a limited knowledge base. For that reason some phrases repeat often in different answers.

Also for sure the used template is imperfect. When experimenting on your own you can try for example to convince chat to talk about subjects that it shouldn’t pick up (ex. in a production ready solution j‑labs chat probably shouldn’t be willing to talk about politics).

There is also a number of technical aspects which were simplified.

Please notice that conversation memory always uses the same memory key, so the code is not ready to operate in multi-thread environments where potentially it maintains conversations with many people. Other aspect which can be optimized is creation of vector database. During every start of the script an in-memory database is created. It requires calculation of embeddings, which is a paid service provided by OpenAI.

LangChain is not limiting programmers to implement solutions only by using OpenAI API so costs can be reduced also by experimenting with different embeddings and models.

Summary

This article provides a guide on how to create a chat-bot leveraging the power of OpenAI’s GPT model with access to updated knowledge by using the LangChain library. It points out the limitations of the current GPT models, particularly their inability to access real-time information. LangChain offers a solution to this problem by allowing connection of Large Language Models (LLMs) to external sources of data.

The article demonstrates the basics of LangChain, explaining how it connects to an external knowledge source, creates embeddings from the knowledge, stores the embeddings in a vector database, and uses user input to retrieve relevant knowledge from the database.

The tutorial focuses on creating a conversational retrieval chain, which is suitable for human conversation and capable of retrieving knowledge from the vector database.