Google AI Inaccurate In 43% Of Finance-Related Searches
Saturday, November 23, 2024
Source: The College Investor
Google AI overviews are misleading or inaccurate in 43% of finance-related searches, according to our study. This is causing consumer confusion, and potentially harming Americans’ finances. It’s especially bad when it comes to tax and financial aid related queries.
We have seen numerous instances across our Facebook Pages and sub-Reddits of users posting misinformation, and when asked about it, they all say they saw it in Search. We’re used to bad information coming from social media sources (looking at you TikTok), but Search was historically a way for users to find the correct information. Until now…
Here’s a recent thread from our Facebook Group where a user believed that she could convert her 529 plan to a Roth IRA in California:
Example of a user misinformed by AI query. Source: The College Investor
What’s Happening: Over the last year Google as rolled out AI-driven answers to search results – called AI Overviews. These results are featured prominently at the top of the search results page. They’ve also been under fire previously for providing dangerous and wrong answers.
Here’s what they look like:
Google search screenshot of query “How Do I Use An ATM” [Screenshot by The College Investor]
It’s important for searchers to realize that Google is simply analyzing the existing web results and trying to “guess” an answer based on what it knows and what it finds. It links to some of the resources it finds on the right side.
However, at the end of the day this information is not vetted by a professional or anyone with knowledge of personal finance. It may not be accurate. Even Google alludes to it with their disclaimer (which was recently added to some finance-related queries):
Google search screenshot. [Screenshot by The College Investor]
We tested 100 personal finance-related queries across multiple areas of personal finance, including banking, credit, investing, insurance, student loans, and financial aid. You can jump to every query we tested below.
Out of 100 searches, we found that Google AI Overviews were correct in 57 instances, and provided misleading or inaccurate information in 43 instances.
We labeled the incorrect answers three ways:
We found the the AI-Overviews were completely incorrect in 12 instances. This included issues such as providing outdated information on products and services and outdated information on student loan repayment plans.
The remaining 31 errors were either misleading results or the results were missing key information.
Generally, the AI Overviews were correct for basic 101-level questions, such as “what is” or “how to”. They had the most correct answers when covering basic personal finance topics, including banking and insurance.
AI Overviews struggled the most with anything that has nuance – which is most of personal finance.That’s harsh, but most of the incorrect answers involved navigating more complex tax topics, investing topics, and student loan topics.
We were surprised to see that AI Overviews had a grasp of recent trending topics, such as the “Chase Glitch”.
But at the same time, it was surprising that the overviews had outdated information for student loan repayment plans, and things like savings and CD rates.
It also appeared that Google placed some type of manual block on some AI overviews related to the ongoing SAVE student loan repayment plan litigation and Biden’s loan forgiveness plan. We believe this because the AI box appeared, but messaging said “not available for this search”. You can see the example here.
We reached out to Google for comment before publication of this article, and they have not yet responded. We will update this with a response if they provide one.
However, a Google spokesperson previously told Business Insider when asked about the earlier issues with misinformation that the “vast majority of AI Overviews provide high-quality information.”
We don’t disagree – the majority of answers in our test were correct. However, when it comes to personal finance, a 40% error rate is not good.
Historically, Google Search has served a key role in helping people navigate their personal finances – especially in light of bad information floating around on social media.
For us, it’s always been important to provide the right information to readers. That’s why we’re frustrated. I would really hate to see someone pay taxes and a penalty because Google AI said they could convert their 529 plan to a Roth IRA (but they live in California, so they can’t).
Or see someone cancel their car insurance because Google doesn’t mentioned it’s required to drive a vehicle – and then they get in an accident or even worse, hurt someone else as an uninsured motorist.
If Google continues to present bad or misinformation about money topics to searchers, not only could it hurt their personal finances, but it could weaken already poor financial literacy in the United States.
We strongly believe that Google should turn off these AI Overviews on finance related topics, especially tax and investment related queries where the outcome can be especially costly for users. Its likely someone at Google will go in and fix the searches we pointed out – but the problem keeps happening. And it keeps getting called out. Eventually someone high up internally needs to say “hey, this is wrong and bad for users.” We hope that happens soon.
All AI overviews were categorized as follows:
These are topics related to general personal finance, such as banking, budgeting, and credit. It was interesting to see timely topics generate AI overviews, such as the Chase Glitch.
Google search screenshot of query “Where do I find my routing number”? [Screenshot by The College Investor]
Google search screenshot of query “How Do I Pay With Cash?” [Screenshot by The College Investor]
Google search screenshot of query “What If I Don’t Pay My Bills?” [Screenshot by The College Investor]
Verdict: Correct
As a side note, notice the image that was used from The Balance without credit or links to their website in the AI overview.
Google search screenshot of query “What is a wire transfer?” [Screenshot by The College Investor]
Google search screenshot of query “What is the debt snowball method?” [Screenshot by The College Investor]
Verdict: Missing Key Information.
We believe that not enough information was provided about interest rates and interest charges that could impact the consumer.
Google search screenshot of query “How Does A Credit Card Work?” [Screenshot by The College Investor]
Google search screenshot of query “How Does A Check Work?” [Screenshot by The College Investor]
Google search screenshot of query “What Was The Chase Glitch” [Screenshot by The College Investor]
Google search screenshot of query “Why Is It Bad To Just Make The Minimum Payment On A Credit Card?” [Screenshot by The College Investor]
Google search screenshot of query “How Is A Credit Score Calculated?” [Screenshot by The College Investor]
Google search screenshot of query “What Is Considered Bad Credit?” [Screenshot by The College Investor]
Verdict: Incorrect
Of course, our opinion is never Wells Fargo. But regardless, there are lots of free checking account options and this list is misleading at best.
Google search screenshot of query “Where To Open A Checking Account?” [Screenshot by The College Investor]
Verdict: Missing Key Information
Since Google AI Overviews pull from sources, these results are missing the actual best rates and accounts. This is a perpetual problem in the savings space, since the big names dominate their lists with paid placements. This isn’t a terrible AI Overview, but definitely missing some of the best savings accounts.
Google search screenshot of query “Where To Open A Savings Account?” [Screenshot by The College Investor]
Verdict: Incorrect
This list is very outdated. There have not been 12-month CD rates over 5% for some time since the Fed has been cutting rates.
Google search screenshot of query “Where To Open A 12-Month CD?” [Screenshot by The College Investor]
Verdict: Missing Key Information
This is a very odd list of places to include, and doesn’t really provide any information about why you should apply at any of these.
Google search screenshot of query “Where Can I Open An Auto Loan Online?” [Screenshot by The College Investor]
These are topics related to taxes. These topics are concerning to us the most because they have large financial implications for incorrect or misleading answers.
Google search screenshot of query “What Is The IRA Contribution Limit?” [Screenshot by The College Investor]
Verdict: Misleading
This AI overview is misleading because it only mentions the lower limits. Roth IRA income limits phase out starting at $146,000 but actually cap at $161,000 for single filers. Married filers cap out at $240,000.
As a side note, the AI overview does get this correct in the Mega Backdoor Overview.
Google search screenshot of query “What Are The Roth IRA Income Limits?” [Screenshot by The College Investor]
Google search screenshot of query “What Is The HSA Contribution Limit?” [Screenshot by The College Investor]
Google search screenshot of query “What Is The 401k Contribution Limit?” [Screenshot by The College Investor]
Verdict: Missing Key Information
This query is missing the key 529 plan annual gift tax exclusion limit. It has a lot of other limits, but missed the most obvious limit that families rely on annually.
Google search screenshot of query “What Is The 529 Plan Contribution Limit?” [Screenshot by The College Investor]
Verdict: Misleading
This AI overview of qualifying 529 plan expenses includes student loan repayment, which is not an eligible expense in all states.
Google search screenshot of query “What Are Qualified 529 Plan Expenses?” [Screenshot by The College Investor]
Verdict: Misleading
The answer to whether you can use a 529 plan for elementary school is “maybe”, depending on your state. The AI Overview leads with “Yes”, which is misleading, however, it does mention that some states may not consider it qualifying in the third bullet point.
Google search screenshot of query “Can You use a 529 plan for elementary school?” [Screenshot by The College Investor]
Verdict: Incorrect
This answer completely ignores that one-third of states (including large states like California and New York) do not allow this. Unlike the elemtary school query, this answer does not include the exceptions in the bullet points. The bottom line is there are very specific 529 plan to Roth IRA rollover rules.
If California residents were to do this, they would faces taxes and penalties.
Google search screenshot of query “Can You Use A 529 Plan For A Roth IRA?” [Screenshot by The College Investor]
Verdict: Correct
However, the second bullet point doesn’t really appy.
Google search screenshot of query “Does California Have A 529 Plan Penalty?” [Screenshot by The College Investor]
Google search screenshot of query “How much do I need to take for an RMD?” [Screenshot by The College Investor]
Google search screenshot of query “Is It Legal To Pay My Child?” [Screenshot by The College Investor]
Verdict: Missing Key Information
The maximium amount you can contribute to a solo 401k is $69,000 technically, the the real amount is very nuanced. The second bullet point alludes to it, but this answer is missing a lot of key information.
Google search screenshot of query “How Much Can I Contribute To A Solo 401k?” [Screenshot by The College Investor]
Verdict: Incorrect
This has been a common TikTok trend, but the reality is the there’s no good reason to own your own house in an LLC, and since it is pass-through, it would not help with taxes. Rather, it would add cost and complication to your life.
Google search screenshot of query “Does Owning Your House In An LLC Help With Taxes?” [Screenshot by The College Investor]
Verdict: Incorrect
While the sentence is correct that you don’t get a Federal tax deduction, 34 states currently offer a tax deduction or tax credit to a 529 plan. This is mentioned in bullet point three, but since the actual sentence is “No”, we view this as incorrect.
Google search screenshot of query “Can I Get A Tax Deduction For A 529 Plan?” [Screenshot by The College Investor]
Verdict: Misleading
An S Corp is simply a tax election for an LLC.
Also, notice that the common “For Financial Advice, Consult A Professional” is not present.
Google search screenshot of query “Should I Open An S Corp Or LLC?” [Screenshot by The College Investor]
Verdict: Incorrect
Since an LLC is a pass-through entity, it has no impact on your taxes. Bullet Point #2 is incorrect, and Bullet Point #5 is incorrect.
Google search screenshot of query “Do I Need An LLC For Rental Property?” [Screenshot by The College Investor]
Verdict: Incorrect
The answer is almost always no.
Google search screenshot of query “Should I Have My Rental Property In An S Corp?” [Screenshot by The College Investor]
These are topics related to investing and investment vehicles.
Verdict: Missing Key Information
The three step process for a backdoor Roth IRA, while correct, misses many parts of the process and doesn’t highlight any pitfalls that get a lot of investors in trouble. Messing up the backdoor Roth IRA can lead to taxes and penalties.
Also, note the infographic from The White Coat Investor, which is not linked in the AI Overview.
Google search screenshot of query “What Is The Backdoor Roth IRA?” [Screenshot by The College Investor]
Verdict: Missing Key Information
This information about the Mega Backdoor Roth IRA is also semi-correct, but misses a lot of key information and pitfalls. As a side note, it does get the IRA contribution limits correct.
Google search screenshot of query “What Is The Mega Backdoor Roth IRA?” [Screenshot by The College Investor]
Verdict: Missing Key Information
This information is correct, but it misses key nuances, such as the fact the wash sale rule doesn’t apply to cryptocurrency, nor retirement accounts.
Google search screenshot of query “What Is The Wash Sale Rule?” [Screenshot by The College Investor]
Verdict: Misleading
You do not have to have earned income to open a Roth IRA, but you do to make the contribution during that year.
Google search screenshot of query “How Do I Open A Roth IRA?” [Screenshot by The College Investor]
Google search screenshot of query “How Do I Open An HSA?” [Screenshot by The College Investor]
Google search screenshot of query “Can I Use A Roth IRA To Pay For College?” [Screenshot by The College Investor]
Verdict: Misleading
While most of this information is accurate, the last bullet point implies that someone needs to have earned income to open a 529 plan. That’s incorrect.
Google search screenshot of query “Can I Invest If I’m Under 18?” [Screenshot by The College Investor]
Google search screenshot of query “How Much Money Do I Need To Start Investing?” [Screenshot by The College Investor]
Google search screenshot of query “What’s The Difference Between Saving And Investing?” [Screenshot by The College Investor]
Google search screenshot of query “Are Municipal Bonds Taxable?” [Screenshot by The College Investor]
Google search screenshot of query “Can You Trade Options In Your IRA?” [Screenshot by The College Investor]
Verdict: Missing Key Information
There are a lot of nuances here that are just missing from this AI overview, and the taxes implications can be harsh.
Google search screenshot of query “Can I Withdraw From My Roth IRA Early?” [Screenshot by The College Investor]
Google search screenshot of query “How Do Expense Ratios Work?” [Screenshot by The College Investor]
Google search screenshot of query “How Do I Invest In Stocks?” [Screenshot by The College Investor]
Verdict: Misleading
Some of this information is not correct – like you typically cannot open an IRA at a life insurance company (unless they do more than insurance). Also, you likely don’t want to open an IRA at a bank.
Google search screenshot of query “Where To Open An IRA?” [Screenshot by The College Investor]
Verdict: Correct
Not a bad list, but honestly missing some of the best brokerage firms. There are also no links for the AI Overview, so it’s unknown where Google is pulling this data from.
Google search screenshot of query “Where To Open A Brokerage Account?” [Screenshot by The College Investor]
Verdict: Missing Key Information
This is not a bad list, but it doesnt’ tell you why you should open an account at any of these places. There’s no fee listing, no account minimums, where does a consumer even start?
Google search screenshot of query “Where To Open An Options Trading Account” [Screenshot by The College Investor]
These are topics related to student loans. It was interesting to see a mix of both outdated information, and good answers to current queries (such as about the pending SAVE litigation).
There also appears to be “blocks” on specific queries that try to generate an AI Overview, but then stop. This is different than simply not showing an AI Overview. Queries related to the SAVE repayment plan and Biden’s plans all deliver this error message:
AI Overview Not Available For This Search. [Screenshot by The College Investor].
Verdict: Misleading
There is no need to apply for federal loans since they are awarded via your financial aid award from filling out the FAFSA. How much in federal loans you’re eligible for depends on your student status (dependent or independent), along with what year of college you’re in.
Google search screenshot of query “How To Get Student Loans?” [Screenshot by The College Investor]
Google search screenshot of query “Do I Have To Pay Back Student Loans?” [Screenshot by The College Investor]
Verdict: Missing Key Information
Yes, you can be fired for student loans under numerous situations, but simply having them is not a reason. This AI overview does mention one reason, but there are many more.
Google search screenshot of query “Can You Get Fired For Student Loans?” [Screenshot by The College Investor]
Verdict: Incorrect
The REPAYE student loan repayment plan no longer exists, it was replaced by the SAVE plan (which is also currently paused). This is all incorrect.
Google search screenshot of query “Can I Apply For The REPAYE Repayment Plan?” [Screenshot by The College Investor]
Verdict: Correct
This is correct, but there is no context in this AI Overview about the pending litigation and problems processing repayment plans.
Google search screenshot of query “Can I Apply For The ICR Repayment Plan?” [Screenshot by The College Investor]
Verdict: Misleading
There are various features here that need to be elaborated on. While the links are helpful, incorrect information in this program, specifically repayment plan options, have been a huge problem.
Google search screenshot of query “Can I Apply For Public Student Loan Forgiveness?” [Screenshot by The College Investor]
Google search screenshot of query “Why Are My Student Loans In Forbearance Due To SAVE?” [Screenshot by The College Investor]
Google search screenshot of query “Does A Processing Forbearance Count For PSLF?” [Screenshot by The College Investor]
Verdict: Correct
We would like to see a little more clarification, but the answer is correct.
Google search screenshot of query “Does Working In A Medical Group Count For PSLF?” [Screenshot by The College Investor]
Verdict: Misleading
You do not have to apply each year, you only need to certify all 120 payments (and you can do this one time if desired). However, it is recommended to submit your PSLF Form annually.
Google search screenshot of query “Do I Have To Apply For PSLF Every Year?” [Screenshot by The College Investor]
Google search screenshot of query “Can I Get Student Loan Forgiveness As A Teacher?” [Screenshot by The College Investor]
Google search screenshot of query “Can I Get Student Loan Forgiveness As A Nurse?” [Screenshot by The College Investor]
Google search screenshot of query “Is Student Loan Forgiveness Taxable?” [Screenshot by The College Investor]
Google search screenshot of query “Is Public Service Loan Forgiveness Taxable?” [Screenshot by The College Investor]
Verdict: Missing Key Information
While you “can” consolidate your student loans at those times, you can also do it other times. And the query was “when”, which implies does it make sense to. Furthermore, the result doesn’t highlight details like PSLF credits, interest capitalization, and access to repayment plans.
Google search screenshot of query “When To Consolidate My Student Loans” [Screenshot by The College Investor]
Google search screenshot of query “Is There A Statute Of Limitations On Student Loans” [Screenshot by The College Investor]
Google search screenshot of query “Can Student Loans Garnish Your Wages?” [Screenshot by The College Investor]
These are topics related to insurance. We discuss life insurance and related products, but also homeowners and other types of insurance.
Verdict: Misleading
The key sentence at the beginning, while factually correct, makes it sound like it is not worth it.
Google search screenshot of query “Is Term Life Insurance Worth It?” [Screenshot by The College Investor]
Google search screenshot of query “When Should I Get Life Insurance?” [Screenshot by The College Investor]
Google search screenshot of query “What Scenarios Should A Young Person Get Life Insurance?” [Screenshot by The College Investor]
Google search screenshot of query “Is Indexed Universal Life Insurance Better Than A 401k?” [Screenshot by The College Investor]
Verdict: Misleading
While it touches on some of the negatives of an IUL, it present it as there is a 50/50 comparison, when the reality is that an IRA is better 99% of the time.
Google search screenshot of query “Is An IUL Better Than An IRA?” [Screenshot by The College Investor]
Google search screenshot of query “Are Annuities Better Than CDs?” [Screenshot by The College Investor]
Google search screenshot of query “Can You Get Renters Insurance For Your Dorm Room?” [Screenshot by The College Investor]
Verdict: Incorrect
The correct answer is “as long as you drive a vehicle, no.” While there may be some options to not have car insurance, most drivers will need it.
Google search screenshot of query “Can You Opt Out Of Car Insurance?” [Screenshot by The College Investor]
Google search screenshot of query “Can You Opt Out Of Homeowners Insurance?” [Screenshot by The College Investor]
Google search screenshot of query “Is Flood Insurance Required?” [Screenshot by The College Investor]
Google search screenshot of query “Is Earthquake Insurance Required?” [Screenshot by The College Investor]
Verdict: Correct
It’s interesting that, compared to other similar queries, they don’t promote a specific brand here.
Google search screenshot of query “Where To Open A Term Life Insurance Policy?” [Screenshot by The College Investor]
Google search screenshot of query “Where To Open An Indexed Universal Life Insurance Policy??” [Screenshot by The College Investor]
Google search screenshot of query “What Is A Life Insurance Policy Illustration” [Screenshot by The College Investor]
Here are some topics related to home ownership and mortgages. We also included topics related to home buying.
Google search screenshot of query “Do I Qualify For An FHA Loan?” [Screenshot by The College Investor]
Google search screenshot of query “Can I Get Rid Of PMI?” [Screenshot by The College Investor]
Verdict: Misleading
It answers a little bit about why it’s misleading in the bullet points, but the answer is “it depends”.
Google search screenshot of query “Do I Have To Have An Escrow Account?” [Screenshot by The College Investor]
Google search screenshot of query “Should I Have A Home Inspection?” [Screenshot by The College Investor]
Verdict: Missing Key Information
This can be a costly mistake to simply state “yes”. Furthermore, some states have non-refundable earnest money.
Google search screenshot of query “Can I Get My Earnest Money Back If I Cancel A Deal?” [Screenshot by The College Investor]
Verdict: Missing Key Information
This is such a vague query that an AI overview probably should not populate. The amount will vary greatly across the United States and ignores deposits that may be required like due diligence money.
Google search screenshot of query “How Much Earnest Money Do I Have To Put Down?” [Screenshot by The College Investor]
Verdict: Missing Key Information
The rules have changed in the last 6 months and none of this is accurate any longer.
Google search screenshot of query “Do I Have To Pay The Sellers Commission?” [Screenshot by The College Investor]
Verdict: Correct
But notice the image that was used and not cited.
Google search screenshot of query “What Is A Home Equity Investment” [Screenshot by The College Investor]
Here are some financial topics relating to paying for college, including about the FAFSA and other financial aid tools.
Verdict: Missing Key Information
The article is missing the nuance that the FAFSA may not open on time on December 1 because of the testing period currently happening.
Google search screenshot of query “When Does The FAFSA Open?” [Screenshot by The College Investor]
Verdict: Incorrect
This article doubled up information from both a martial arts weapon and a financial aid tool. But even the definition of Student Aid Index is not complete.
Google search screenshot of query “What Is An SAI?” [Screenshot by The College Investor]
Verdict: Misleading
This information is inaccurate as it pulls from various sources that estimate what four years of specific colleges (not even the most expensive) would cost if they were full paid for out of pocket. It appears these results add in room and board and other costs.
The correct list of the most expensive colleges is here, based on tuition.
Google search screenshot of query “What Is The Most Expensive College In The United States?” [Screenshot by The College Investor]
Verdict: Incorrect
Based on the previous query, we wanted to get a more specific answer based on the colleges with the highest tuition. Again, the answer provided by the AI Overview was incorrect and simply provided a similar result. Here is the correct list of colleges with the highest tuition. Kenyon College should be the most expensive, and it’s not on the list.
The numbers provided in this AI Overview aren’t even accurate.
Google search screenshot of query “Which Colleges Have The Highest Tuition?” [Screenshot by The College Investor]
Google search screenshot of query “How Does Financial Aid Work?” [Screenshot by The College Investor]
Google search screenshot of query “Do You Have To Pay Back Grants?” [Screenshot by The College Investor]
Google search screenshot of query “Can I Get An FSA ID Without My Parents?” [Screenshot by The College Investor]
Verdict: Misleading
While the answer does break down the criteria for filing independent for FAFSA, we see this as one of the biggest mistakes that families make filing the FAFSA. The answer is “No” for most undergraduate students.
Google search screenshot of query “Can I File As Independent For FAFSA?” [Screenshot by The College Investor]
Google search screenshot of query “Does A 529 Plan Hurt My FAFSA?” [Screenshot by The College Investor]
Google search screenshot of query “Does a Grandparent 529 Plan Count For FAFSA” [Screenshot by The College Investor]
Verdict: Missing Key Information
The last bullet point is the most important – a Roth IRA is NOT reported on the FAFSA, but income from it can count.
Google search screenshot of query “Does A Roth IRA Count For FAFSA?” [Screenshot by The College Investor]
Verdict: Misleading
The CSS Profile does count retirement accounts like the Roth IRA in their own separate area, and schools may use this information.
Google search screenshot of query “Does A Roth IRA Count For CSS Profile?” [Screenshot by The College Investor]
Google search screenshot of query “Where Do I Get Scholarships” [Screenshot by The College Investor]
Google search screenshot of query “Where Do I Get Grants To Pay For College?” [Screenshot by The College Investor]
These AI overviews were generated on October 15 and 16, 2024. We focused on major questions in each personal finance topic area, along with trends of questions we’ve seen being asked on social media.
As extreme weather becomes increasingly frequent and severe, its disproportionate impact on low- and moderate-income (LMI) communities grows. The recent...
(Bloomberg) — One of the most divisive elections in recent US history is prompting some Americans to consider moving overseas. Interest in...
Like so much of our culture, the word “gig” comes to us via New Orleans jazz. The exact origin is...
