Generate an LLM evals reference file of the papers in the database

[sahilds1]

[sahilds1]

I generated an LLM evals reference file of the papers that I had in my local database using the below code snippet:

from sqlalchemy import create_engine
import pandas as pd

engine = create_engine("postgresql+psycopg2://balancer:balancer@localhost:5433/balancer_dev")
# Filter out papers that shouldn't be used from local database
query = "SELECT * FROM api_embeddings WHERE date_of_upload > '2025-03-14';"
df = pd.read_sql(query, engine)

df['formatted_chunk'] = df.apply(lambda row: f"ID: {row['chunk_number']} | CONTENT: {row['text']}", axis=1)
# Ensure the chunks are joined in order of chunk_number by sorting the DataFrame before grouping and joining
df = df.sort_values(by=['name', 'upload_file_id', 'chunk_number'])
df_grouped = df.groupby(['name', 'upload_file_id'])['formatted_chunk'].apply(lambda chunks: "\n".join(chunks)).reset_index()
df_grouped = df_grouped.rename(columns={'formatted_chunk': 'concatenated_chunks'})
df_grouped.to_csv('~/Desktop/formatted_chunks.csv', index=False)

We can use this to generate an LLM evals reference file for all the papers in the prod database

[sahilds1]

I was able to generate an LLM evals reference file for all the papers in the prod database with the backup .sql file

# Create a new PostgreSQL database named "backupDBBalancer07012025"
createdb backupDBBalancer07012025
# Restore a PostgreSQL database from a backup `.sql` file into the newly created database
pg_restore -v -d backupDBBalancer07012025 ~/Downloads/backupDBBalancer07012025.sql

# Create a SQLAlchemy engine to connect to the restored PostgreSQL database
# The connection string format is: postgresql://<username>@<host>:<port>/<database>
engine = create_engine("postgresql://sahildshah@localhost:5432/backupDBBalancer07012025")