RAGAS for RAG app evaluation (with aws bedrock)

pip install -U llama-index llama-index-core
pip install langfuse datasets ragas llama_index python-dotenv openai --upgrade
pip install ragas
TRY 2
pip install pyarrow==16.1.0
pip install nltk
pip install ragas
pip install boto3

In credentials file have below

[default]
aws_access_key_id=abc
aws_secret_access_key=xyz

Do point to below credentials

export AWS_SHARED_CREDENTIALS_FILE='/Users/sid/credentials'

Usage

# data Load
from datasets import load_dataset

amnesty_qa = load_dataset("explodinggradients/amnesty_qa", "english_v2")

# Import required metrics
from ragas.metrics import (
    context_precision,
    faithfulness,
    context_recall,
)

from ragas.metrics.critique import harmfulness

# list of metrics we're going to use
metrics = [
    faithfulness,
    context_recall,
    context_precision,
    harmfulness,
]


# Pointing to AWS Bedrock instead of OpenAI
config = {
    "credentials_profile_name": "default",
    "region_name": "ap-south-1",  # E.g. "us-east-1"
    "model_id": "anthropic.claude-3-sonnet-20240229-v1:0",  # E.g "anthropic.claude-v2"
    "model_kwargs": {"temperature": 0.4},
}

bedrock_model = BedrockChat(
    credentials_profile_name=config["credentials_profile_name"],
    region_name=config["region_name"],
    endpoint_url=f"https://bedrock-runtime.{config['region_name']}.amazonaws.com",
    model_id=config["model_id"],
    model_kwargs=config["model_kwargs"],
)

# init the embeddings
bedrock_embeddings = BedrockEmbeddings(
    credentials_profile_name=config["credentials_profile_name"],
    region_name=config["region_name"],
)

Running the evaluation

from ragas import evaluate

result = evaluate(
    amnesty_qa["eval"].select(range(3)),
    metrics=metrics,
    llm=bedrock_model,
    embeddings=bedrock_embeddings,
)

# export the scores
df = result.to_pandas()
df.to_csv('score.csv', index=False)
df.head()

Results are shown below

                                question                                       ground_truth                                             answer  ... context_recall  context_precision  harmfulness
0  What are the global implications of the USA Su...  The global implications of the USA Supreme Cou...  The global implications of the USA Supreme Cou...  ...            1.0           1.000000            0
1  Which companies are the main contributors to G...  According to the Carbon Majors database, the m...  According to the Carbon Majors database, the m...  ...            1.0           1.000000            0
2  Which private companies in the Americas are th...  The largest private companies in the Americas ...  According to the Carbon Majors database, the l...  ...            1.0           0.333333            0

[3 rows x 8 columns]

References:

Ragas official

Sample content for data_samples