We Tested AI on Credit Card Travel Insurance: How & Why It Failed

Posted on December 3, 2024 by Alexey Saltykov

Background: AI and Credit Card Travel Insurance

copilotandchatgpt.png

AI is widely used in many industries to handle tasks like analyzing and creating content, and the travel industry is no different.

While AI has great potential, it’s important to be cautious. In some cases, using AI without care can do more harm than good.

At TravelAndCards.com, our platform supports the travel industry—including travel agencies, advisors, aggregators, suppliers—and travel insurance companies/brokers by providing high-quality data on travel insurance associated with credit cards. This data is invaluable for:

  1. Increasing travel insurance sales, e.g., by addressing customer objections regarding travel insurance on credit cards.
  2. Streamlining travel insurance claims, e.g., by accurately determining claim volumes split between primary and secondary travel insurance providers.

With the rise of AI-powered tools such as ChatGPT and Microsoft Co-Pilot, many travel professionals wonder if these solutions can simplify the process of finding detailed information about travel insurance on credit cards.

To address this question, we tested these AI solutions to evaluate the quality of their results.

Below, we share several scenarios where we asked AI straightforward questions. You can try them yourself to verify the results if you're curious.

AI Test Scenario 1:

Question: “How many days does travel medical insurance on BMO Ascend Wolrd Elite Credit Card cover?”

MICROSOFT CO-PILOT AI response: INCORRECT

wronganswer.pngThe BMO Ascend World Elite Mastercard provides emergency medical coverage for up to 31 days during your trip. If you need coverage for a longer trip, you may be able to purchase additional coverage.

ai-and-travel-insurance-test1-copilot

 

ChatGPT AI response: INCORRECT

wronganswerThe BMO Ascend World Elite Mastercard provides travel medical insurance for trips lasting up to 21 days for cardholders under the age of 65. For those aged 65 or older, the coverage period is reduced to 3 days. This benefit applies to cardholders and their eligible family members traveling with them, offering peace of mind during short-term trips. For more details, check the card's specific terms and conditions.

ai-and-travel-insurance-test1-chatgpt

 

CORRECT TEST ANSWER AS PER BANK TRAVEL INSURANCE POLICIES OR TRAVELANDCARDS.COM:

Travel Medical Insurance

Covers you, your spouse and/or dependent children for up to $5 million in eligible Out-of-Province/Out-of-Country Emergency Medical expenses for trips up to 21 days.

Source: https://www.bmo.com/main/personal/credit-cards/bmo-ascend-world-elite-mastercard/

AI Test Scenario 2:

Question: “What is the age cut-off for travel medical insurance on BMO Ascend World Elite Travel Credit Card?”

MICROSOFT CO-PILOT AI response: INCORRECT

wronganswerThe BMO Ascend World Elite Mastercard provides travel medical insurance for cardholders, their spouses, and dependent children who are under the age of 75.

ai-and-travel-insurance-test2-copilot

 

ChatGPT AI response: INCORRECT

wronganswer

The BMO Ascend World Elite Mastercard does not have an age cut-off for eligibility for its travel medical insurance. However, the duration of coverage depends on the cardholder's age:

  • Under 65 years old: Covered for up to 21 days per trip.
  • 65 years or older: Covered for up to 3 days per trip.

This ensures cardholders receive appropriate short-term travel medical coverage based on their age. For full details, refer to the insurance policy document provided by BMO.

ai-and-travel-insurance-test2-chatgpt

 

CORRECT TEST ANSWER AS PER BANK TRAVEL INSURANCE POLICIES OR TRAVELANDCARDS.COM:

Out-Of-Province/Country  Emergency Medical Insurance

Twenty-one (21) consecutive days of coverage per trip, if you are age sixty-four (64) or under on your departure date.

AI Test Scenario 3:

Question: “Does BMO eclipse Visa Infinite card have trip cancellation coverage?”

MICROSOFT CO-PILOT AI response: INCORRECT

wronganswerYes, the BMO eclipse Visa Infinite Privilege Card does offer Trip Cancellation Insurance. This coverage allows for reimbursement of prepaid, non-refundable travel expenses if you need to cancel your trip for covered reasons.

ai-and-travel-insurance-test3-copilot

 

CORRECT TEST ANSWER AS PER BANK TRAVEL INSURANCE POLICIES OR TRAVELANDCARDS.COM:

There is no trip cancellation insurance coverage associated with this card. The AI misinterpreted the card's name and provided information for the BMO Eclipse Visa Infinite Privilege Card, which was not the card in question. This mistake can be difficult for an agent to detect.

AI Test Scenario 4:

Question: “What are coverage limits for Trip Cancellation on Manulife Money Visa Infinite Card?”

MICROSOFT CO-PILOT AI response: INCORRECT

wronganswerThe Manulife Money Visa Infinite Card provides Trip Cancellation Insurance coverage up to $3,500 per insured person per trip, with an overall maximum limit of $6,000 per policy for Multi-Trip All-Inclusive Plans.

ai-and-travel-insurance-test4-copilot

 

ChatGPT AI response: INCORRECT

wronganswerThe Manulife Money+ Visa Infinite Card offers trip cancellation insurance with a maximum coverage limit of $2,000 per person, up to a total of $5,000 for all covered persons on the same trip. This benefit helps protect against financial losses if a trip is canceled for an eligible reason before departure.

ai-and-travel-insurance-test4-chatgpt

 

CORRECT TEST ANSWER AS PER BANK TRAVEL INSURANCE POLICIES OR TRAVELANDCARDS.COM:

There is no trip cancellation insurance included with this card.

Why Did AI Fail to Respond Correctly to These Questions?

There are several reasons for AI’s shortcomings in this context. While AI excels at finding, consolidating, and summarizing information that is relatively static and widely known - for example:

  • Describing natural phenomena
  • Answering geographical questions
  • Summarizing key elements of philosophical theories
  • Translating text between languages
  • Performing calculations or estimates

 

It struggles significantly in the following situations though:

  • When data is spread across multiple sources
  • When data is stored in various formats (e.g., online webpages, PDFs)
  • When the information changes frequently over time

 

This is particularly true for credit card travel insurance data, which fits all these challenges. Based on our observations, AI often makes the following mistakes:

  • Using outdated online PDFs to extract information
  • Pulling incorrect data from third-party websites (instead of official banking sources)
  • Confusing credit card names due to their similarity
  • Mixing up details about coverage included with a credit card versus standalone travel insurance products

These limitations highlight the difficulty AI faces when handling complex, dynamic, or fragmented information.

Final Word on Using AI for Analyzing Credit Card Travel Insurance Information

In our experience, with over a decade in the travel insurance space, nothing can replace thorough human data review and assessment for this type of information. That’s why hundreds of travel organizations across Canada trust TravelAndCards.com and rely on our services.

In our opinion, using AI for credit card travel insurance analysis often causes more harm than good, and we strongly advise against it. The potential consequences may be far too costly.

Interested in trying out our platform for travel advisors to increase travel insurance sales? Register here for free – the first month is on us.

Leave a Reply

This comment form is under antispam protection

David Burnie

This is an illuminating article. The AI responses seem so confident in its responses. And if you don't look closely with a strong knowledge of the cards (e.g., in scenario 3 where "privilege" is added) then you'll be sharing incorrect information. Thanks for pointing out the challenges in using AI in serving travel clients.
1 week ago

Alexey Saltykov

In reply to by David Burnie (not verified)

Exactly. Many users, however, tend to over-rely on AI and blindly trust its responses. As our assessment has demonstrated, AI can make significant mistakes that may have dangerous consequences, particularly in the context of travel insurance. We remain strong proponents of human-assisted data review and moderation.
2 hours 11 minutes ago