Climbing the AI Mountain

My Journey with LLMs and Data Integration
Climbing AI Mountain image

Image generated by OpenAI’s DALL-E 3 model for this blog.

I’ve been working on integrating LLM (Large Language Model) functionality into our platform, and learning a lot along the way…often at the cost of sleep. The journey has been filled with valuable lessons and amusing moments. As many organizations strive to enable similar functionalities, I hope sharing my lessons learned and discussing some of the considerations for implementation will be beneficial and entertaining.

One funny lesson involved my first attempt to add a PDF copy of Frankenstein to be referenced by a locally hosted LLM to question on the PDF (using Retrieval-Augmented Generation setup that I will go into more detail farther in this post). Picture this: I was working late into the night during a thunderstorm, and after many failed attempts, I finally succeeded in getting my code to process the PDF and have it referenced by the LLM to answer questions. With maniacal laughter, I reveled in satisfaction when I finally got responses from my stitched-together solution.  However, I soon found myself between elation and disappointment. While I had managed to get responses based on parts of the story, they were so broken up and so hard to understand that I couldn’t see how they related to the questions asked. Somehow, using Frankenstein, I had unknowingly turned Meta’s $500 million+ Llama-3 model into Frankenstein’s monster. Not quite what I wanted it to learn from the story. Talk about creating a monstrous misunderstanding!

As I continued climbing the AI knowledge mountain, I often encountered complex challenges. Each time, I would spend hours learning and experimenting, only to discover an Open Source project that had already built something more reliable and easier to run, which greatly helps getting a project together. The Open Source communities working together in this space have been amazing. They not only provided tools and solutions but also a sense of camaraderie. Many people helped by answering problems I ran into, and I soon found myself able to help others who encountered issues I had resolved only a few days earlier.

Data Integration Techniques for Large Language Models

LLMs have emerged as powerful tools capable of understanding and generating human-like text; however, it is crucial to employ effective data integration techniques to maximize their potential. Here are some of the things I’ve learned about data integration for LLMs. Please let me know if you have any other options you are considering or if you see I have something wrong. I feel most situations will be using a combination of all of these techniques in some form or another, the challenges are determining the best one for each situation.


Think of embeddings as translating your documents into a ‘language’ that the LLM can easily understand and quickly reference. They are mathematical representations of information in a high-dimensional space that store data of all types, including text, images, audio files, and documents. By creating embeddings for your documents, databases, and other data sources, we enable the LLM to find and relate information quickly. This method is efficient for handling large datasets and improves the accuracy of responses.


  • Fast and efficient data retrieval
  • Improved relevance and accuracy of responses


  • Requires substantial computational resources for large datasets
  • Initial setup can be complex

Some top Open Source tools for embeddings include txtai, an all-in-one embeddings database, and Chroma, an AI-native embedding database that integrates well with other tools like LangChain and OpenAI.

Retrieval-Augmented Generation (RAG):

RAG combines retrieval-based techniques with generative models. It retrieves relevant documents or data points and uses them to generate contextually accurate responses. This approach enhances the LLM’s ability to provide detailed and specific answers. When used properly, it should minimize the number of hallucinations in responses, but will not entirely prevent them.


  • Provides highly accurate and context-specific responses
  • Combines the strengths of retrieval and generation
  • Can be cost effective for small amounts of data


  • Requires a robust retrieval system
  • Can be computationally intensive for large amounts of data

Some effective Open Source tools for RAG include RAG on Hugging Face Transformers, RAGFlow, and Deepset Haystack.

Fine-Tuning a Model:

Fine-tuning involves training the LLM on a specific dataset related to your domain. By fine-tuning, the model learns the nuances of your data, resulting in more relevant responses tailored to your needs. This is beneficial if you work in an area where the use of vocabulary is specific to your industry, such as the legal and medical fields.


  • Highly customized responses
  • Improved model performance for specific tasks


  • Requires labeled training data and computational resources
  • Time-consuming to train and update

Popular Open Source models for fine-tuning include xTuring, OpenLLM, and H2O LLM Studio. These tools provide comprehensive frameworks for customizing and fine-tuning various LLMs.

Extended Context Windows:

LLMs typically have a fixed context window, limiting the amount of text they can process simultaneously. Extended context windows allow models to consider larger chunks of text, improving their ability to understand long documents and to maintain coherence in the generated text. Techniques such as attention mechanisms and hierarchical models can be used to extend context windows, enabling LLMs to handle more complex tasks.


  • Better understanding of complex and detailed queries
  • More coherent and contextually accurate responses


  • Increased computational load
  • Potential for slower response times

Data Types for Large Language Models

Integrating various types of data effectively is crucial to maximizing the potential of LLMs. Here are most of what we all have to deal with. Below are the most commonly accessed types of data and the challenges they present. You will find different sources might have some structured, unstructured data, and semi-structured data to deal with.

Unstructured Data:

Unstructured data, such as text files, PDFs, and web pages, presents a challenge for LLMs due to its lack of predefined format. Effective handling of unstructured data involves parsing, cleaning, and organizing the data into a structured format that LLMs can process.

The platform provides powerful tools for transforming unstructured data into a structured format ready for LLM applications. Their Open Source libraries support a wide range of document types, making it easier to preprocess data for RAG and other application types. In our test environments this tool has been particularly useful for me pre-processing data for LLMs.

Structured Data:

Structured data, like that found in spreadsheets and databases, is already organized and so easy for LLMs to process. This data type includes numerical and categorical information that can be used for various analytical tasks.

Internet Searches:

Integrating data from internet searches provides real-time access to the latest information. This method uses search APIs to pull relevant data from the web and make it accessible to LLMs in order to generate accurate and up-to-date responses. Some webpages aren’t as friendly for web crawling processing so when you are feeding it directly into an active context window you can get some unexpected results.

Code Repositories:

Code repositories contain valuable information for technical tasks. Integrating this data involves programmatic processes created for pulling code snippets, comments, and documentation from repositories to assist LLMs in generating code-related responses. Each area can use separate procedures to process to make sure it is referenced correctly with the right context.

User-Generated Content:

User-generated content, such as forum posts and social media updates, provides insights into public opinion and trends. This data type requires careful handling with the correct workflows to ensure privacy and relevance in responses.


APIs offer a way to access structured and semi-structured data (data that is a cross between structured and unstructured data, containing some structure but not fitting into a traditional database schema) from various services. Integrating data from APIs allows LLMs to pull information from different platforms, enhancing their ability to provide comprehensive

What do we do with all this?

For our planned integration and desired functionality, we’re dealing with multiple Open Source applications, each with different databases; and we are creating unstructured and structured data in various formats. We have data repositories with code bases, constantly modified and created user content, and various APIs. Figuring out the best ways to integrate this data while maintaining verification of authorship, protecting data privacy, securing the LLMs from prompt injections, preventing sharing protected information, and managing resource concerns is an ongoing challenge. By employing these latest methods, we ensure our AI-enhanced productivity platform remains at the forefront of innovation and provides robust, accurate, and secure solutions for our users.

These efforts align with the growing need for every company that wants to stay competitive in the evolving market to become a technology company. By building on advanced infrastructure and leveraging these solutions, we help users build innovative solutions without needing deep technical knowledge. This allows organizations to be continuously innovative, creating tailored, powerful tools and services that cater to specific business needs, while enhancing their potential to be more scalable, efficient, sustainable, and forward-looking.

We still have plenty to figure out regarding the details, but feel we are making good progress. We can’t wait to share more with everyone!

I’d love to hear about your experiences and challenges in integrating diverse data sources with LLMs. What methods have you found most effective? Share your thoughts and join the conversation!

Skip to content