Share or save this article

Share on X

AUDIO

Listen to this article

3:35

日本語

Checking your sign-in status…

An audio edition based on this article.

We ask AI to revise our writing, write code, or solve a difficult problem. Once the task is finished, we feel we have done it.

But could we do the same thing the next day with AI switched off?

Two things that seem similar are different: **results achieved with AI** and **skills we can apply without AI**. Comparing studies published in 2025–2026 shows evidence that AI helps people work. It also highlights easily overlooked problems when people hand over all the work involved in learning. This article asks: **How can we assess and develop our own abilities while getting help from AI?**

First, measure completion and mastery separately

Suppose AI corrects the grammar and word choices in an English writing assignment. The submitted text may be better. But that does not necessarily mean the writer can choose those words when writing a new piece on a different topic.

This is not unique to AI. Consulting an answer key may help someone finish a problem without learning how to solve it. The difference is that AI makes revisions so smoothly and quickly that **it is easy to mistake task completion for evidence of learning**.

We therefore distinguish between two measures.

| What is measured | Examples | What it reveals | |---|---|---| | Results achieved while using AI | Quality of deliverables, time spent, number of problems solved | What was achieved during the task | | Skills after AI is removed | Solving a different problem independently, explaining errors | What someone can do without assistance |

This distinction also helps us interpret research findings accurately.

An experiment with 52 developers: a clearer gap in understanding than speed

A 2026 study by Shen and Tamkin involved 52 developers using the Python programming language. Participants wrote code with a tool new to them called Trio. Trio is a library—a collection of components for use in programs—that provides ways to handle waits during processing. Participants were randomly assigned to two groups: 26 used AI assistance and 26 did not.[1]

After the task, participants were tested without AI on their ability to understand how the code worked, read it, and fix errors. **On a comprehension test scored out of 27, the AI group's average was 4.15 points lower than that of the group without AI.** The AI group finished faster on average, but the main analysis did not find a statistically clear reduction in task time.[1]

AI users did not all score poorly. The researchers classified AI use into six patterns. They reported that understanding was preserved in three patterns that retained the thinking process, such as thinking through the task while asking AI for explanations.[1]

However, the differences between the six patterns were not tested by randomly assigning participants to usage styles. **The study did not establish that a particular way of asking AI reliably causes learning.** The experiment also involved a specific development task, so its findings cannot be applied directly to every job or age group. The paper is a preprint—research made public before peer review.[1]

Experiments with 1,222 participants: what happened after they got the answers

In another 2026 study, Liu and colleagues reported several randomized experiments involving a total of 1,222 participants. They examined performance on tasks such as fraction calculations and reading comprehension, both with AI assistance and after it was removed.[2]

In the first experiment, after AI assistance was removed, the AI group answered **57%** of fraction problems correctly, compared with **73%** in the group that had not used AI. The AI group also skipped **20%** of problems, compared with **11%** in the non-AI group.[2]

| First experiment: after AI assistance was removed | AI group | Non-AI group | What the difference means | |---|---:|---:|---| | Share answered correctly | 57% | 73% | 16 percentage points lower in the AI group | | Share skipped without solving | 20% | 11% | 9 percentage points higher in the AI group |

Note: Values are group averages converted to percentages. **They are not pooled results from a single test involving all participants.** The table shows only the first experiment. The paper also noted potential bias from that experiment's exclusion criteria and attrition. Subsequent experiments made improvements to address these concerns.[2]

Another analysis in the same study found an interesting distinction. People who **asked AI for answers directly** tended to perform worse and show less persistence after assistance was removed, with different results for those who **asked for hints**.[2] However, comparisons between usage styles may reflect differences arising from participants' own choices. These findings do not establish that using only hints guarantees improvement.

Here, “persistence” refers to behavior measured during the experiment, such as whether participants skipped problems, rather than a personality trait across everyday life. A short experiment cannot establish how years of AI use might change people's abilities.

Counter-evidence: newcomers can also learn from AI at work

It would be premature to conclude from this that people are better off avoiding AI.

A study by Brynjolfsson, Li, and Raymond, published in The Quarterly Journal of Economics in 2025, analyzed work data from 5,172 customer support agents. After agents gained access to AI-generated response suggestions, the number of issues resolved per hour **increased by an average of 15%**. Gains were particularly large among less experienced agents.[3]

The AI suggested responses that agents could send as written, edit, or ignore. The authors discussed the possibility that this made it easier for newcomers to adopt the response patterns of experienced colleagues.[3]

At first glance, this appears to contradict the two earlier studies. But they measured different things. The customer support study focused on **performance in real work**, while the programming study focused on **whether participants could understand new concepts on their own**. The customer support analysis also used differences in when AI was rolled out within a company. Its design was not exactly the same as that of the learning experiments.[3]

Even so, this is important counter-evidence. AI is more than simply “a tool that takes away people's ability to think.” Depending on its design and use, it can also bring knowledge to people with less experience.

Three studies show where AI use can go wrong

These studies examined different jobs and tasks, so their effects cannot be added together to say that “AI is X% better overall.” Instead, a common question is **which parts of the process you handle yourself**.

**Steps suitable for delegation to AI** are often those where saving time is itself the goal and you can check the finished work. By contrast, **steps to keep for yourself** involve practicing the skills you want to develop or exercising the judgment needed to spot AI errors.

For example, if you need to send an email in English right away, having AI polish it may be reasonable. But if you want to learn to write in English independently, you need to draft it yourself first, examine the reasons for the revisions, and try applying them to another sentence before you can see progress.

Even with the same AI, the right way to use it changes with the goal. Forgetting this distinction can lead you to mistake “a way to finish faster” for “a way to improve.”

Two records to start keeping tomorrow

What follows is **FORWARD BASIS's proposal**, rather than a learning method whose effectiveness these studies directly demonstrated. To improve how you use AI, record results in two columns.

| Record | What to record each time | Example assessment | |---|---|---| | Today's results | Time spent using AI, quality of the finished work, number of errors | “Cut time to delivery by 40 minutes” | | Tomorrow's skills | Whether you could answer similar questions without AI and explain your reasoning | “Solved 3 out of 5 similar questions independently” |

The procedure is simple. First, briefly write down your own thinking. When consulting AI, ask for reasons and alternatives as well as answers. Finally, close AI and solve **a similar question that memorizing the previous answer will not help you answer** on your own. Trying again the next day makes it easier to see whether you are relying solely on what you remember from the session.

However, the three studies discussed here did not establish the optimal interval for testing yourself after closing AI, including whether the next day is best. To assess effectiveness, you also need conditions for comparison in your own records, such as practice without AI and questions matched for difficulty.

FORWARD BASIS's conclusion: Add another measure for evaluating AI

**The question is what to delegate to AI and what to keep for yourself, rather than whether to use AI at all.** This is the central judgment connecting the studies discussed here.

Finishing work faster with AI has value. There is also evidence that it helps newcomers in the workplace. But when the goal is learning, success while using AI alone cannot establish whether skills have improved. You need to measure whether that performance can be reproduced without assistance.

FORWARD BASIS's own framework is to **measure “output quality” and “the ability to reproduce that performance without assistance” on separate scales**. The former reflects today's work; the latter reflects tomorrow's capabilities. Only by checking both can you distinguish when to use AI as a shortcut and when to deliberately think for yourself.

SOURCE

Sources

These are the primary sources we used to check the facts. The article identifies our own analysis.

  1. Sourcehttps://arxiv.org/abs/2601.20245
  2. Sourcehttps://arxiv.org/abs/2604.04721
  3. Sourcehttps://doi.org/10.1093/qje/qjae044
NEXT BASIS
More in Personal Development →
Personal Development2026.11.11About 5.8 min read READ

Cal Newport Prioritizes When Work Ends Over Rising Early—His Approach to Time Management

Drawing on Cal Newport’s own records from 2008–2022 and behavioral science research, this article examines the method of setting a workday end time first, then deciding which work to keep or drop—and the method’s limits.

Business2026.11.14About 7.6 min read READ

Revenue Down 12%, Operating Profit Up 49%: Why Console Unit Sales Alone Cannot Explain Nintendo’s Business

Nintendo’s interim financial results on November 5, 2026, showed that profit rose even as console unit sales fell. However, software’s share of sales was not the only factor. Understanding the profit increase requires distinguishing its drivers, including a one-time factor: US tariff refunds.

Business2026.11.14About 5.3 min read READ

Does Costco’s 10% Revenue Growth Mean Customers Bought 10% More? Separating Gasoline, Exchange Rates, and Gross Profit

In fiscal 2026, Costco increased net sales from merchandise and other sales by about 10% from the previous year. That looks like solid performance for a membership warehouse retailer. But revenue reflects more than the volume customers bought. It also reflects gasoline prices, exchange rates used to convert overseas revenue into dollars, and the impact of adding stores. When revenue grows, what should we conclude has increased? We break down the company’s official US financial results.[1][2]