Skip to main content

Talk about AI tends to swing between hype and dismissal. A citizen's view needs to be more specific: which jobs are these tools good at, which are they bad at, and how can you tell the difference before you rely on them?

General-purpose tools

Earlier AI systems were narrow. One program played chess, another read postal codes on envelopes. Today's large models are general-purpose, meaning one model handles many kinds of tasks without separate training for each. The same chatbot can draft a cover letter, translate a menu, explain a lease, write a spreadsheet formula and suggest names for a cat.

Where they shine

Language models are strongest at tasks built from patterns in text: drafting, summarizing, rewording for a different audience, translating between widely used languages, brainstorming options and writing or explaining routine computer code. Image systems can sort photos, read handwriting and flag possible problems in medical scans for a doctor to review. These are tasks where a good first attempt is valuable and where a person can judge the result.

Some of the most important uses are in science. Proteins fold into complex shapes that determine what they do, and working out a single shape once took years of laboratory work. AlphaFold, built by Google DeepMind, predicts a protein's three-dimensional shape from its sequence of amino acids, and in 2022 its makers released predicted structures for more than 200 million proteins. In 2024 Demis Hassabis and John Jumper shared half the Nobel Prize in Chemistry for this work. The other half went to David Baker for designing new proteins.

The jagged frontier

The catch is that ability is uneven in ways that are hard to predict. In a 2023 field experiment, researchers from Harvard Business School and several other universities gave 758 consultants at the Boston Consulting Group realistic tasks, some with access to GPT-4 and some without. On tasks inside the AI's range, consultants using it finished about 12 percent more work, about 25 percent faster, with answers judged roughly 40 percent better. One task, though, was designed to sit just outside that range: it required careful reading of interview notes and figures that contradicted a surface impression. There AI users were less likely to reach the right answer than consultants working alone, by about 19 percentage points, because many accepted the tool's confident but mistaken analysis.

The authors called this pattern a jagged frontier: AI's abilities are uneven, strong on some tasks and weak on others that look similar. A model may write a fine summary of a contract and then miscount the words in it, or solve a hard math problem and fumble a simple riddle with a twist. The frontier does not follow a smooth line between easy and hard as a person would judge them.

Reading benchmark claims

Companies report scores on benchmarks, which are standard tests in math, coding, science or law. These are useful but imperfect. One known problem is benchmark contamination: test questions leak into the training data and inflate the scores, much as a student who has seen the answer key looks smarter than they are. Another is that a test measures only what it measures. Passing the multiple-choice part of a bar exam is not the same as representing a frightened client.

Capabilities also change quickly. A sweeping statement that AI cannot do something, or can now do something, should carry a date and a source, because abilities can change within months, and a confident judgment from two years ago may now be wrong in either direction. The examples in this course are accurate as of 2026, and some will age.

A rule of thumb

The practical lesson is to match the tool to the task. AI fits best where its output is cheap to check and mistakes are cheap to fix: a first draft you will revise, a list of ideas you will filter, code you can test, a translation a fluent speaker will review. It fits worst where errors are costly and hard to spot, such as an unverified medicine dose, a court filing, or a factual claim you will repeat to thousands of people.

Worked example: the food bank flyer

A volunteer at a small food bank needs a flyer in English, Spanish and Vietnamese announcing new hours. She asks a chatbot for a short, friendly draft and two translations, and in two minutes she has all three. The English version is easy for her to check, so she edits it herself. She cannot check the Vietnamese, so she asks a parent who visits the food bank to read it over. He fixes one stiff phrase.

Next she asks the chatbot how many families the food bank served last year. It offers a number, but she realizes it has no way to know: that figure lives in the food bank's own records. Then she pastes in a state grant's eligibility rules and asks for a summary. The summary is helpful, but she checks the two rules that decide whether they qualify against the original text. In one afternoon she used the tool on both sides of the frontier, and the skill that mattered most was knowing which side each task was on.

Quiz

5 questions · Practice: scored, not saved

1. Why should a claim about what AI can or cannot do carry a date?
2. In the 2023 study of consultants, what happened on the task built to lie outside the AI's range?
3. According to the level's rule of thumb, where does AI fit best?
4. Who won the other half of the 2024 Nobel Prize in Chemistry?
5. What does the jagged frontier describe?

CivilSimian.com created by AxiomaticPanic, CivilSimian, Kalokagathia