How to merge vector stores?
Hi. I’m building an LLMops workflow using the RAG approach.
For RAG, I built a vector store with the FAISS Vector Store Creator node using a single pdf file, and for continued expansion, I plan to continue creating pdf files as vector stores to add to this workflow.
At this time, in order to continuously update new PDF files in one vector store, I must go through the process of reloading the files that were previously created in the vector store, merging them with the new files, and creating a new vector store.
The picture below is a workflow that attempted to use Concatnate by transforming the vector store into tool form. However, this method also requires the process of loading the existing vector store, and as the capacity of the existing vector store increases, loading becomes more difficult.
Instead of doing this, is there a way to append only the vectorized data of a new pdf file to the existing vector store, like appending a csv file with the same name in KNIME?
I believe this approach is the most efficient for continued scalability of LLMops. There is code available in Python, but it would be nice if it could be done with a KNIME node.