Hospitable.com: Evaluating and Optimizing Personalized Message Replies for Airbnb Hosts

The short-term rental market continues to grow, with platforms like Airbnb and Vrbo driving high occupancy rates for hosts. However, this success brings challenges: managing repetitive, time-consuming tasks such as guest communication, check-ins, cleaning schedules, and listing updates across platforms. These operational burdens divert hosts from focusing on property improvements and delivering exceptional guest experiences, which are now necessary in the face of increasing competition in the market. Hospitable.com addresses these challenges by automating the routine tasks of short-term rental management, giving hosts back their time. With features like automated messaging rules, a unified inbox, synced calendars, and task management tools, Hospitable.com reduces administrative strain and lowers the risk of errors like double bookings or missed communications.

One of the AI features that Hospitable.com offers is an inbox reply assistant that generates an AI draft message written on behalf of a short-term rental host to a specific guest on an API-connected platform such as Airbnb. This feature - called “Suggest with AI” on Hospitable’s platform - gets its prompt injected with multiple forms of context to generate a message relevant to that guest’s booking at that point in time, including:

Property data (listing description, available amenities)
Booking information (check-in/out times, inquiry/pending/accepted/canceled status)
Guest information (name, number of guests, pets, coming from, AI-generated summary)
Host information (name, city)
The message history in that conversation between the host and the guest so far

All of this context is already in the Hospitable database instantly fetchable by some combination of unique identifiers relevant to the conversation being looked at. On top of this, there is an additional RAG layer, combining:

Imported property guidebook content (policies, operating instructions, things to do, places to eat, etc) from uploaded PDFs and other online sources
Previous replies from the host to other guests asking similar questions at the same booking stage (pre-booking, pre-checkin, staying)

The AI assistant also gets a set of guidelines as part of the prompt that it should follow when crafting the message, covering:

Tone of voice
Language (English, etc)
Message length
Guardrails (both general and product-specific)
Assumptions

Thousands of B2B customers use Hospitable’s Suggest with AI feature 250,000 times every month.

Despite the advantages of being able to generate relevant and accurate replies in seconds, integrating LLMs comes with its own set of challenges:

Hallucination: LLMs may generate responses that are plausible but factually incorrect, potentially leading to misinformation or confusion in guest communications.
Guideline Adherence: Ensuring that AI-generated content adheres to host, short-term rental platform, and Hospitable policies is critical to avoid both compliance issues and hosts losing revenue or incurring unwanted costs.
Tone of Voice: Maintaining a consistent and appropriate tone in communications is essential for brand integrity.

Hospitable’s strategy uses Patronus AI's evaluation framework to perform the following:

Frequent prompt iteration with A/B tests
Analytics monitoring AI performance based on user actions and conversions
Prompt result QA and smoke testing
In-app customer feedback
Customer interviews and Town Hall

Hospitable is able to ship AI confidently and increase system performance with automated evaluations on the Patronus AI platform. Given known scenarios and data snapshots, prompts are re-run and compared against previously accepted generations. The comparison is done using multiple evaluators looking at distinct aspects of the AI-generated messages that are relevant for Hospitable use case. This results in a CI testsuite for AI, automatically showing both regressions and improvements in output quality!

Tackling these challenges is especially important as Hospitable looks towards the rollout of a fully automated inbox feature in 2025. Patronus AI enables Hospitable.com to roll out AI agents at scale with millions of AI generations for rental guests.