Yes, that's true because for every first query the vector store is created which takes few secs to form and retrieve the answer, for next query same vector store is used this is called lazy loading and it is actually good practice , hence not a bug but a deliberate.

Observed latency is very huge, and this is not a good practice to the customer side, kindly act on this reduce the latency