By: Christopher Bronner, Senior Machine Learning Scientist
February 27, 2024
Business Context: The Knot’s Marketplace and Style Quiz
The Knot is a leading wedding Vendor Marketplace in the U.S. that connects more than 1 million couples with over 200,000 local wedding professionals, such as reception venues, florists, and photographers. We offer a comprehensive suite of planning tools that includes personalized wedding websites, invitations and registry services that make planning easier.
Wedding Vendor Marketplace
The Knot Vendor Marketplace is a central component of the site and we monetize this product through paid subscriptions that allow vendors to purchase listings in more prominent parts of our site.
From the vendor perspective, the value of our Marketplace product is rooted in the leads they receive from couples who can contact vendors directly on our site. Therefore, the rate of couples who submit a lead to a vendor (i.e. conversion to lead submission) is one of our most important metrics to assess the performance of our product.
Conversion to lead submission, in turn, depends on our ability to recommend to couples vendors that are relevant to their personal preferences, including their taste and style. Improving the vendor recommendations can be assumed to increase conversion to lead submission.
Personalization via Style Quiz
In order to make high-quality, personalized recommendations that speak to the couple’s individual needs, we first need information about their taste and style. This is where our “Style Quiz” comes in, which is one of our most popular planning tools.
In the Style Quiz, couples rate images from four galleries (reception venues, table settings, bouquets and dessert decorations) and based on this information, we categorize their style by showing a result consisting of a set of primary and secondary style terms. For example, we might categorize a given couple as having primarily a “classic” style, but also leaning toward “elegant” and “traditional” (secondary styles). In addition to the primary and secondary style, we determine the ideal setting for their wedding—for example, a hotel, loft or barn.
These results give valuable insight into the couple’s individual style preferences, especially for their reception venue. We therefore set out to leverage this information to improve the quality of our vendor recommendations (specifically for reception venues) for our couples. This in turn should lead to an increase in conversion to lead submission which benefits our subscribing vendors.
Methodology: Semantic Search based on Style Quiz Results
Problem definition. We are looking for items (venues) that best match a set of search terms (the result of the Style Quiz). For each venue, several attributes are available for consideration: venue description text, amenities tags, market, guest capacity and price range
Our goal is to identify the venues that are the best match for a couple, based on the results they got from our Style Quiz.
To that end, we applied a semantic search approach that finds venues whose description texts best match the couple’s style terms. More specifically, we represent venues and style terms as embeddings, quantify their semantic similarity with a distance measure in the embedding space, and recommend venues similar to the couple’s individual style.
Semantic Search
The overall concept is to represent venues on the one hand, and the couple’s style terms on the other hand as embeddings (high-dimensional numerical vectors) that encode the semantic meaning of the respective texts, and use a distance metric in the embedding space to quantify the semantic similarity of any pair of venue and style. Recommendations are then made by selecting venues with the highest similarity score.
We started by creating embeddings for the description texts of all venues in our Marketplace. In addition to the descriptions which are written by the vendors themselves, we also included textual meta-information in the embeddings such as amenities provided by the vendor on their storefront (e.g. that they offer a dressing room or wireless internet).
The next step is to encode a couple’s individual style in the same embedding space as the venue descriptions. We do this by creating embeddings for all style terms generated by our Style Quiz—meaning the primary style term, secondary style terms, and setting terms. We then determine an overall style embedding as the linear combination of these embeddings with the coefficients being weights that represent the importance of each of the style terms. Putting it differently, the overall style is the center of gravity of the different styles in the embedding space.
Conceptual illustration of semantic search for wedding venues based on couples’ style. Embeddings of venue descriptions (gray) encode the style of a venue (e.g. rustic vs. modern venues). Recommendations (blue) are generated by finding venues close to the overall style of a couple (red).
Having represented the venues and the couple’s style with embeddings, it’s straightforward to make venue recommendations for a couple: after filtering venues according to a few rules that implement our business logic (e.g. filter for venues in a specific geographic area and considering whether venues are paid or unpaid, or whether the storefront includes a photo), we simply recommend the venues closest to the couple’s overall style in the embedding space.
The distance between two embeddings is calculated as the inner product of the two embedding vectors, which is a good approximation to Euclidean distance in a high-dimensional space, but less computationally expensive.
Match Score
Our semantic search approach for venue recommendations intrinsically provides us with a quantitative measure for the quality of each recommended venue, namely the distance between the venue and the couple’s overall style in the embedding space.
We surface the distance to couples, albeit after inverting and scaling it. The resulting “match score” is shown next to each recommended venue as “94% match,” for example. We hypothesize that this match score highlights to couples that the recommendation is personalized, ultimately increasing conversion to lead submission.
LLM-Generated Recommendation Reasonings
To further highlight to couples the fact that the recommendations are personalized and based on their individual style quiz results, we added LLM-generated copy that provides a reasoning for why a specific recommended venue matches the couple’s style. The screenshot below shows a tested variation of our venue recommendations that features the reasoning copy.
Screenshot from a variant of The Knot’s Style Quiz results page showing a venue recommendations carousel with one highlighted venue. The match score (92%) and the reasoning copy are shown for that venue.
Experimentation and Results
Measuring the performance of a style-based recommendation system is difficult because whether a recommended venue matches a specific style is highly subjective. Ultimately, the only reliable way to assess the impact of our tactics is through A/B tests.
We conducted different A/B tests for the different tactics mentioned above to quantitatively attribute the impact to each change separately.
Test of Semantic Search-Based Venue Recommendations
We exposed 50% of couples with an impression of the Style Quiz results page to the new semantic search-based recommendations. At the same time, the remaining 50% served as a control group and were given recommendations based on an older methodology that didn’t take the style quiz result into account.
The couples in the variant group that were shown the semantic search-based recommendations had a significantly higher conversion rate to lead submission than the couples in the control group. By using semantic search, we achieved a 5.9% lift in this metric.
Test of Recommendation Reasonings
We conducted a second, multivariate test in which one variant featured an updated design of the Style Quiz results page that included the recommendation reasonings among other design updates. This variation led to a lift in conversion to lead submission of 4.1% but was ultimately not rolled out for different considerations.
Test of Match Score
A different variation in the same multivariate test included the match score. The addition of the match score alone led to a statistically significant 4.2% increase in conversion to lead submission. Together with the 5.9% lift from the semantic search-based venue recommendation, this means an overall lift of 10.3%.
Conclusion
The quality of product recommendations depends on their relevance to the preferences of the individual customer. By basing our recommendations on our couples’ choices from the Style Quiz, we can suggest products in a highly personalized manner that we wouldn’t be able to with just behavioral analytics, especially for new members immediately after sign-up.
We find that instead of just serving recommendations by themselves, it adds value for our customers to have additional data points that explain why the recommendations are a good fit for them, namely the match score and the reasoning copy.