TestsetGenerator -> RuntimeError: ... got Future <..> attached to a differen t loop

[abetatos]

This is related to: #681

Describe the bug
TestGeneration launches an error with default example of https://docs.ragas.io/en/stable/concepts/testset_generation.html with VertexAI models.

Ragas version: 0.1.7
Python version: Python 3.10.13

Code to Reproduce

generator_llm = VertexAI(model_name="text-bison", temperature=0, response_validation=False)
critic_llm = VertexAI(model_name="gemini-pro", temperature=0, response_validation=False)
vertexai_embeddings = VertexAIEmbeddings("textembedding-gecko@001")

generator = TestsetGenerator.from_langchain(
    generator_llm,
    critic_llm,
    vertexai_embeddings
)

distributions = {
    simple: 0.5,
    multi_context: 0.4,
    reasoning: 0.1
}

loader = PubMedLoader("liver", load_max_docs=10)
documents = loader.load()

testset = generator.generate_with_langchain_docs(documents, 10, distributions, with_debugging_logs=True) 
testset.to_pandas()

Error trace

Filename and doc_id are the same for all nodes.                                                                                                                        
Generating:   0%|                                                                                                                               | 0[/10](http://localhost:8889/10) [00:00<?, ?it[/s](http://localhost:8889/s)][ragas.testset.filters.DEBUG] context scoring: {'clarity': 3, 'depth': 3, 'structure': 3, 'relevance': 3, 'score': 3.0}
[ragas.testset.evolutions.DEBUG] keyphrases in merged node: ['Hyperuricemia', 'Protein peptides', 'Anti-hyperuricemia properties', 'Uric acid-lowering peptides', 'Functional foods']
[ragas.testset.filters.DEBUG] context scoring: {'clarity': 3, 'depth': 3, 'structure': 3, 'relevance': 3, 'score': 3.0}
[ragas.testset.evolutions.DEBUG] keyphrases in merged node: ['Liver surgery', 'Liver anatomy and physiology', 'Liver resection', 'Vascular control', 'Liver transplantation', 'Residual liver function']
[ragas.testset.filters.DEBUG] context scoring: {'clarity': 2, 'depth': 2, 'structure': 2, 'relevance': 3, 'score': 2.25}
[ragas.testset.evolutions.DEBUG] keyphrases in merged node: ['Ammonia sensor', 'Colorimetric optical readout', 'Plasmonic Ag[/SiO](http://localhost:8889/SiO) nanoparticles', 'Animal serum', 'Paper-based sensor']
[ragas.testset.filters.DEBUG] context scoring: {'clarity': 1, 'depth': 1, 'structure': 1, 'relevance': 1, 'score': 1.0}
[ragas.testset.evolutions.INFO] retrying evolution: 0 times
[ragas.testset.filters.DEBUG] context scoring: {'clarity': 3, 'depth': 3, 'structure': 3, 'relevance': 3, 'score': 3.0}
[ragas.testset.evolutions.DEBUG] keyphrases in merged node: ['Liver surgery', 'Liver anatomy and physiology', 'Liver resection', 'Vascular control', 'Liver transplantation', 'Residual liver function']
[ragas.testset.filters.DEBUG] context scoring: {'clarity': 3, 'depth': 3, 'structure': 3, 'relevance': 3, 'score': 3.0}
[ragas.testset.evolutions.DEBUG] keyphrases in merged node: ['Hyperuricemia', 'Protein peptides', 'Anti-hyperuricemia properties', 'Uric acid-lowering peptides', 'Functional foods']
[ragas.testset.filters.DEBUG] context scoring: {'clarity': 3, 'depth': 2, 'structure': 3, 'relevance': 3, 'score': 2.75}
[ragas.testset.evolutions.DEBUG] keyphrases in merged node: ['Metastatic breast cancer', 'Age groups', 'Metastatic patterns', 'Survival outcomes', 'Elderly patients']
[ragas.testset.filters.DEBUG] context scoring: {'clarity': 3, 'depth': 2, 'structure': 3, 'relevance': 3, 'score': 2.75}
[ragas.testset.evolutions.DEBUG] keyphrases in merged node: ['Metastatic breast cancer', 'Age groups', 'Metastatic patterns', 'Survival outcomes', 'Elderly patients']
[ragas.testset.filters.DEBUG] context scoring: {'clarity': 2, 'depth': 2, 'structure': 2, 'relevance': 3, 'score': 2.25}
[ragas.testset.evolutions.DEBUG] keyphrases in merged node: ['Ammonia sensor', 'Colorimetric optical readout', 'Plasmonic Ag[/SiO](http://localhost:8889/SiO) nanoparticles', 'Animal serum', 'Paper-based sensor']
[ragas.testset.filters.DEBUG] context scoring: {'clarity': 3, 'depth': 2, 'structure': 3, 'relevance': 3, 'score': 2.75}
[ragas.testset.evolutions.DEBUG] keyphrases in merged node: ['Metastatic breast cancer', 'Age groups', 'Metastatic patterns', 'Survival outcomes', 'Elderly patients']
[ragas.testset.filters.DEBUG] context scoring: {'clarity': 3, 'depth': 3, 'structure': 3, 'relevance': 3, 'score': 3.0}
[ragas.testset.evolutions.DEBUG] keyphrases in merged node: ['Liver surgery', 'Liver anatomy and physiology', 'Liver resection', 'Vascular control', 'Liver transplantation', 'Residual liver function']
Generating:   0%|                                                                                                                               | 0[/10](http://localhost:8889/10) [05:20<?, ?it[/s](http://localhost:8889/s)]

Exception in thread Thread-31:
Traceback (most recent call last):
  File "/Users/.../miniconda3/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "/Users/.../venv/lib/python3.10/site-packages/ragas/executor.py", line 96, in run
    results = self.loop.run_until_complete(self._aresults())
  File "/Users/.../venv/lib/python3.10/site-packages/nest_asyncio.py", line 98, in run_until_complete
    return f.result()
  File "/Users/.../miniconda3/lib/python3.10/asyncio/futures.py", line 201, in result
    raise self._exception.with_traceback(self._exception_tb)
  File "/Users/.../miniconda3/lib/python3.10/asyncio/tasks.py", line 232, in __step
    result = coro.send(None)
  File "/Users/.../venv/lib/python3.10/site-packages/ragas/executor.py", line 84, in _aresults
    raise e
  File "/Users/.../venv/lib/python3.10/site-packages/ragas/executor.py", line 79, in _aresults
    r = await future
  File "/Users/.../miniconda3/lib/python3.10/asyncio/tasks.py", line 571, in _wait_for_one
    return f.result()  # May raise f.exception().
  File "/Users/.../miniconda3/lib/python3.10/asyncio/futures.py", line 201, in result
    raise self._exception.with_traceback(self._exception_tb)
  File "/Users/.../miniconda3/lib/python3.10/asyncio/tasks.py", line 232, in __step
    result = coro.send(None)
  File "/Users/.../venv/lib/python3.10/site-packages/ragas/executor.py", line 38, in sema_coro
    return await coro
  File "/Users/.../venv/lib/python3.10/site-packages/ragas/executor.py", line 112, in wrapped_callable_async
    return counter, await callable(*args, **kwargs)
  File "/Users/.../venv/lib/python3.10/site-packages/ragas/testset/evolutions.py", line 142, in evolve
    ) = await self._aevolve(current_tries, current_nodes)
  File "/Users/.../venv/lib/python3.10/site-packages/ragas/testset/evolutions.py", line 298, in _aevolve
    results = await self.generator_llm.generate(
  File "/Users/.../venv/lib/python3.10/site-packages/ragas/llms/base.py", line 92, in generate
    return await agenerate_text_with_retry(
  File "/Users/.../venv/lib/python3.10/site-packages/tenacity/_asyncio.py", line 142, in async_wrapped
    return await fn(*args, **kwargs)
  File "/Users/.../venv/lib/python3.10/site-packages/tenacity/_asyncio.py", line 58, in __call__
    do = await self.iter(retry_state=retry_state)
  File "/Users/.../venv/lib/python3.10/site-packages/tenacity/_asyncio.py", line 110, in iter
    result = await action(retry_state)
  File "/Users/.../venv/lib/python3.10/site-packages/tenacity/_asyncio.py", line 78, in inner
    return fn(*args, **kwargs)
  File "/Users/.../venv/lib/python3.10/site-packages/tenacity/__init__.py", line 410, in exc_check
    raise retry_exc.reraise()
  File "/Users/.../venv/lib/python3.10/site-packages/tenacity/__init__.py", line 183, in reraise
    raise self.last_attempt.result()
  File "/Users/.../miniconda3/lib/python3.10/concurrent/futures/_base.py", line 451, in result
    return self.__get_result()
  File "/Users/.../miniconda3/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/Users/.../venv/lib/python3.10/site-packages/tenacity/_asyncio.py", line 61, in __call__
    result = await fn(*args, **kwargs)
  File "/Users/.../venv/lib/python3.10/site-packages/ragas/llms/base.py", line 177, in agenerate_text
    result = await self.langchain_llm.agenerate_prompt(
  File "/Users/.../venv/lib/python3.10/site-packages/langchain_core/language_models/llms.py", line 643, in agenerate_prompt
    return await self.agenerate(
  File "/Users/.../venv/lib/python3.10/site-packages/langchain_core/language_models/llms.py", line 1018, in agenerate
    output = await self._agenerate_helper(
  File "/Users/.../venv/lib/python3.10/site-packages/langchain_core/language_models/llms.py", line 882, in _agenerate_helper
    raise e
  File "/Users/.../venv/lib/python3.10/site-packages/langchain_core/language_models/llms.py", line 866, in _agenerate_helper
    await self._agenerate(
  File "/Users/.../venv/lib/python3.10/site-packages/langchain_google_vertexai/llms.py", line 413, in _agenerate
    res = await _acompletion_with_retry(
  File "/Users/.../venv/lib/python3.10/site-packages/langchain_google_vertexai/llms.py", line 123, in _acompletion_with_retry
    return await _acompletion_with_retry_inner(
  File "/Users/.../venv/lib/python3.10/site-packages/tenacity/_asyncio.py", line 142, in async_wrapped
    return await fn(*args, **kwargs)
  File "/Users/.../venv/lib/python3.10/site-packages/tenacity/_asyncio.py", line 58, in __call__
    do = await self.iter(retry_state=retry_state)
  File "/Users/.../venv/lib/python3.10/site-packages/tenacity/_asyncio.py", line 110, in iter
    result = await action(retry_state)
  File "/Users/.../venv/lib/python3.10/site-packages/tenacity/_asyncio.py", line 78, in inner
    return fn(*args, **kwargs)
  File "/Users/.../venv/lib/python3.10/site-packages/tenacity/__init__.py", line 390, in <lambda>
    self._add_action_func(lambda rs: rs.outcome.result())
  File "/Users/.../miniconda3/lib/python3.10/concurrent/futures/_base.py", line 451, in result
    return self.__get_result()
  File "/Users/.../miniconda3/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/Users/.../venv/lib/python3.10/site-packages/tenacity/_asyncio.py", line 61, in __call__
    result = await fn(*args, **kwargs)
  File "/Users/.../venv/lib/python3.10/site-packages/langchain_google_vertexai/llms.py", line 121, in _acompletion_with_retry_inner
    return await llm.client.predict_async(prompt, **kwargs)
  File "/Users/.../venv/lib/python3.10/site-packages/vertexai/language_models/_language_models.py", line 1399, in predict_async
    prediction_response = await self._endpoint.predict_async(
  File "/Users/.../venv/lib/python3.10/site-packages/google/cloud/aiplatform/models.py", line 1643, in predict_async
    prediction_response = await self._prediction_async_client.predict(
  File "/Users/.../venv/lib/python3.10/site-packages/google/cloud/aiplatform_v1/services/prediction_service/async_client.py", line 404, in predict
    response = await rpc(
  File "/Users/.../venv/lib/python3.10/site-packages/google/api_core/grpc_helpers_async.py", line 85, in __await__
    response = yield from self._call.__await__()
  File "/Users/.../venv/lib/python3.10/site-packages/grpc/aio/_call.py", line 299, in __await__
    response = yield from self._call_response
  File "/Users/.../miniconda3/lib/python3.10/asyncio/futures.py", line 285, in __await__
    yield self  # This tells Task to wait for completion.
RuntimeError: Task <Task pending name='Task-78' coro=<as_completed.<locals>.sema_coro() running at /Users/.../venv/lib/python3.10/site-packages/ragas/executor.py:38> cb=[as_completed.<locals>._on_completion() at /Users/.../miniconda3/lib/python3.10/asyncio/tasks.py:558]> got Future <Task pending name='Task-795' coro=<UnaryUnaryCall._invoke() running at /Users/.../venv/lib/python3.10/site-packages/grpc/aio/_call.py:568>> attached to a different loop

If you need any additional info please let me know!
Thank you very much.

[abetatos]

I forgot to add that the error was tested both in a jupyter notebook (using nest_asyncio.apply()) and a .py.

And the imports were both tested with

from langchain_google_vertexai import VertexAIEmbeddings, VertexAI

and

from langchain.chat_models import ChatVertexAI
from langchain.embeddings import VertexAIEmbeddings

[franchukpetro]

Same problem for me with VertexAI stack in jupyter notebook. Slightly different models and embeddings used, but I'm getting this error right from the start of test set generation process.

[franchukpetro]

@abetatos UPD

I've take a look into this PR, as a result of issue you've mentioned here.

I tried to clone that dev branch with fix, and install ragas from that source, but that hasn't worked.
Then, I tried just to replace the src/ragas/executor.py file contents with the fixed code, presented in that PR, in my local (master) ragas package, and that worked - I'm able to generate test datasets with the VertexAI stack.

I still don't understand what is the root cause, and whether that fix is legit (since repo author hasn't really answered and merged that PR), but at least that works.