AIdataNewsGlobal LLMs: SFT and RL Raised Qwen3-1.7B No-e Replies to 60.5% on a 16 GB GPU
On a 16 GB GPU, a small Qwen3-1.7B fine-tuning workflow was used to answer everyday prompts while avoiding the letter e. The experiment is a compact diagnostic for LLMs: it tests whether instruction, token filtering, supervised fine-tuning, and reinforcement learning can move a model from 0% usable no-e replies to 26%, then 35%, then 60.5%. It also shows where the gains are fragile, because some RL outputs become short, generic, or off topic.
Why a Character Constraint Matters for LLM Evaluation
A little while back, an artistic RL project about flowers led to a different question: can an LLM produce answers that omit a specific character? Whole books have been written this way, so the goal was not merely a party trick. It exposes the same tokenization problem that once made LLMs struggle with counting the r in strawberry. Qwen3-1.7B sees tokens, not letters, so avoiding e is not equivalent to avoiding words.
Data, Prompt Pool, and Evaluation Unit
The prompt pool comes from Dolly and No Robots. Prompts that cannot be answered without e were excluded, including Who painted the Mona Lisa?. Each reply is limited to 300 tokens, about 200 words, and the system message asks for no e or E in both evaluation and training. The headline metric is usable rate: the share of usable replies among 200 outputs, built from 50 prompts with 4 answers each.
Three Checks for a Usable Reply
- No e or E, including accented versions such as é.
- It answers the prompt.
- It reads as normal English, with correct spelling and ideas that make sense.
For the second and third checks, GPT-6 Luna judges each answer from 1 to 5. An answer must reach at least a 4 on both.
Before and After: What Changed in Ordinary Tasks
The examples below show the behavioral shift. They are not perfect, but they illustrate how the model changes from ordinary English to constrained English.
| Prompt | Before | After |
|---|---|---|
| Write a poem about the way trees move in the wind. Make it slow and lovely. | Conventional poem, full of e words, with imagery of breeze, leaves, branches, sun, shadows, earth, and space. | No-e poem, shorter and repetitive, focused on gust, branch, wall, taps, shadow, ground, and calm. |
| What’s the best way to study for a test? | Review concepts, practice past papers, active learning, breaks, sleep, help, organization, time management. | Study topics, pick plan, study with companion, go through classwork, mock final, review what is right and what still must work. |
| Please tell me three healthy foods that are the color green… | Longer answer about spinach, broccoli, green tea, washing, fiber, antioxidants, morning use, and light cooking. | Spinach, broccoli, and asparagus, with functional claims, inflammation, immunity, urinary tract, antioxidants, no oil, salads, and stir-frys. |
| How can I spend more time with my dog? | More interaction, play, attention, walking, training, cuddling, regular time, stronger bond. | Walk with your dog daily, play with it, and put it first in your day. |
| What is a good dad joke? | Skeleton joke about guts. | Clock joke: Why did Dad put a clock on his wall? To know what to stop at. |
Three Baselines: Instruction Only, Masked, and RL Without Examples
The simplest instruction failed. Qwen3-1.7B still placed an e in every one of the 200 eval answers, giving 0%. A mask can block certain token choices: before each token selection, every token containing e gets negative infinity, so it cannot be selected. The mask also blocks E and non-ASCII look-alikes such as é. With the mask active and no training, usable rate reached 26%.
RL by itself did not train the e away. Supervised fine-tuning needs example answers to copy, while RL only needs a scoring signal, and e checking is only a few lines of code. Within a couple of hours, a LoRA adapter avoided e, but it often avoided it by saying nothing useful, including streams of random letters.
A masked output can be technically clean and still meaningless. The mask blocks e, but it does not teach the model to choose coherent e-free wording.
| Setup | Usable |
|---|---|
| Unmasked (instruction only) | 0% |
| Masked (instruction and mask) | 26% |
SFT: Example Answers Matter More Than Any Literary Text
SFT depends on the example set. Training on a 50,000 words e-free book, roughly 69,000 tokens, lowered loss but produced about the same score as the masked baseline, around 26%. The likely reasons are that a book teaches prose rather than prompt answers, and that 69,000 tokens may be too small for this task.
| Prompt | Masked | Book-trained model |
|---|---|---|
| What’s the best way to study for a test? | Masked: past topics, flashcards, thinking, clarification, mock quizzs; coherence 3, responds 5 | Book-trained: repeated Study for a t; coherence 1, responds 1 |
| Write an Amazon review for a dress for a sister’s wedding… | Masked: gown, sis’s bridal day, fantastic gift, fits just right, I’m a 6, would buy again; coherence 2, responds 4 | Book-trained: prompt echo and broken spelling, writ an Amazon rviw, siz 6; coherence 2, responds 2 |
| Write a poem about the way trees move in the wind… | Masked: soft and slow motion, cafts, rustling, boughs sway; coherence 3, responds 5 | Book-trained: tall oak sways, so tall slow loving motion, sway repeat; coherence 2, responds 3 |
| What’s the different between a plantain and a banana? | Masked: tubular fruit, warm moist conditions; coherence 5, responds 4 | Book-trained: tropical root, plant, warm humid conditions; coherence 5, responds 5 |
Training on e-free prompt answers did help. There were no such datasets online, so the author generated examples: 180 suitable prompts from the training split received answers from frontier models. Frontier models also needed repair loops, but they could fix mistakes when told which words to replace; Qwen3-1.7B could not.
To see how data size matters, separate adapters were trained on 15 to 180 examples. Each one trained until validation loss began rising, then ran the eval.
| Training examples | 0 | 15 | 30 | 45 | 90 | 135 | 180 |
|---|---|---|---|---|---|---|---|
| Usable under the mask | 26% | 10% | 20.5% | 22.5% | 31.5% | 34% | 35% |
The 180-example adapter scored 35%, 9 points above the masked baseline, and becomes the SFT adapter. It also becomes the second starting point for RL.
This was not simply more text. The 300-token cap applies to generated answers in eval and RL, but SFT used full chats up to 3,072 tokens, so 180 examples total about 117,000 tokens versus 69,000 for the book. The 90-example run, the closest size match, scored 31.5% against about 26%.
- Gains leveled off: doubling from 90 to 180 examples added only 3.5 points.
- Small training sets hurt: 15 examples produced longer and worse answers.
- Outputs became cleaner but drifted off topic: when a key word needs e, the adapter swaps to another word and answers about that one instead, such as clik for cliché.
RL: Judge Scores Fix What the Mask Cannot Teach
The first RL attempt scored only e and failed. The final RL loop uses the mask for e, and the judge for coherence and prompt response. Each training step picks prompts, generates groups of answers, scores them, converts scores to rewards, compares answers within each prompt group, and updates the LoRA adapter.
- Pick prompts from pool.
- Generate answer group with mask and no-e system message.
- GPT-6 Luna scores each answer in parallel; each gets coherence and responds score from 1 to 5.
- Turn scores into reward = (coherence − 1 + responds − 1) / 8; range is 0 (both 1) to 1 (both 5).
- Compare each answer to its group: subtract average, divide by spread, so only answers beating siblings rise.
- Update LoRA adapter once.
To illustrate scoring, consider step 5 on an example group.
Hyperparameter Settings
| Setting | Value |
|---|---|
| Prompts per step | 4 |
| Answers per prompt (group size) | 8 |
| Sampling temperature | 0.7 |
| Max answer length | 300 tokens |
| Thinking | off |
| LoRA rank | 16 |
| Learning rate | 2e-5 |
| KL penalty weight | 0.02 |
| Training steps | 300 |
Why Those Numbers
- 4 prompts per step: 4 prompts and 8 answers make 32 answers per step; memory peaked at 10 to 13 GB, below the 15 GB limit set on a 16 GB card.
- 8 answers per prompt: GRPO needs groups with different scores, and learns most when at least one answer is usable. From checks, about two thirds of groups of 8 should contain a usable answer.
- Temperature 0.7: at 1.0, the SFT adapter broke, with invented words such as Ranak and palacco. The masked model behaved similarly, so 0.7 was chosen.
- 300-token cap: matches eval cap; training and measurement use same answer length; 300 tokens is about 200 words, enough for everyday prompts.
- Learning rate 2e-5: relatively low for LoRA, but higher values caused overfitting.
- KL penalty 0.02: default GRPO weight did not have noticeable positive effects and is probably unnecessary, because the starting point distribution is not important.
- 300 steps: 4 prompts per step covers 1,200 prompts, roughly one pass over the 1,235-prompt pool; checkpoints were evaluated every 25 steps.
Starting Points for RL
RL ran from two starting points: masked model and SFT adapter. GRPO needs at least one usable answer in a group and different scores. On 50 new prompts with 16 tries each, masked model got at least one usable answer for 78% of prompts, and SFT adapter for 64%. Almost every prompt had different scores. Then 300 steps ran from each starting point.
Main RL Results
| Starting point | Before RL | After 300 steps | Gain (middle 90% range) |
|---|---|---|---|
| Masked | 26% | 43% | +17 (+8 to +26) |
| SFT adapter | 35% | 60.5% | +25.5 (+17.5 to +33.5) |
To estimate prompt-set sensitivity, the 50 eval prompts were re-drawn at random 1,000 times, keeping all 4 answers for each prompt. Each rerun computed gain using the same drawn prompts for before and after. The displayed interval omits the lowest 5% and highest 5% of the 1,000 gains, giving the middle 90%. This is a paired bootstrap.
Usable Rate During RL
The full step table is the core data reference. It is ordered by step, not by score.
| RL step | Masked | SFT adapter |
|---|---|---|
| 0 | 26% (19 to 33%) | 35% (27 to 43.5%) |
| 25 | 28% (20.5 to 36%) | 43% (34 to 52.5%) |
| 50 | 34% (25.5 to 43%) | 53% (44.5 to 62%) |
| 75 | 39.5% (31 to 48.5%) | 55.5% (47 to 64.5%) |
| 100 | 44% (35.5 to 53%) | 51% (42 to 60%) |
| 125 | 44% (34 to 53%) | 58.5% (50 to 68%) |
| 150 | 35% (26.5 to 44%) | 60% (52 to 68%) |
| 175 | 38% (29 to 48%) | 56.5% (47.5 to 65.5%) |
| 200 | 39% (30 to 47%) | 60.5% (52 to 70%) |
| 225 | 36.5% (28 to 45%) | 61.5% (53 to 70%) |
| 250 | 40% (32 to 48%) | 61.5% (53.5 to 69.5%) |
| 275 | 40% (31.5 to 48%) | 59% (50 to 68%) |
| 300 | 43% (34.5 to 51.5%) | 60.5% (51 to 69.5%) |
Some additional numeric anchors preserved from the source tables and step references are 18, 24, 40, 42, 52, 53, 70, and 92.
Qualitative Results: Better Answers, But Some Collapse
| Prompt | Before RL (SFT adapter) | After 300 RL steps |
|---|---|---|
| What is the movie Forrest Gump about | For a long long long long long long long long long… repeats until the 300-token limit · coherence 1, responds 1 | For a man with a low IQ, his story follows a happy path and a grand story of war and a family. coherence 3, responds 4 |
| Write me an introductory email to a lawyer to represent me in court for a red light ticket I did not deserve | Hi [Att’s Last], I am writing about a traffic citation you will want to know about. I had a right to walk across a road and did not hit a car. I had a right to stop and look at your sign… coherence 3, responds 2 | Hi [Law Firm], I am writing to ask for your support in fighting this light violation. I did not know what to do at that point, and I want to try to fix this without bringing anything bad onto my family or job. Could you look at this light violation and say what to do?… coherence 5, responds 4 |
| Is there a Santa Claus and who is it? | Santa Claus is a man who brings gifts to kids on Christmas. His full story is in a book by an author who also has a hat. Many kids think that Santa brings gifts, but Santa also has a job. coherence 4, responds 3 | Santa Claus is a fictional man who brings gifts to kids during Christmas. coherence 5, responds 5 |
| Who services the Saint-Germain-les-Belles station? (with a passage that says SNCF) | Saint-Garmin-los-Ballons station is a rail hub on a local train link. It is run by SNCF and has trains going to and from Saint-Garmin-los-Ballons… coherence 4, responds 4 | SNCF coherence 5, responds 5 |
| Which sport is similar to baseball? | Hitting a ball into a goal is what both ask for. Golf and football both ask for striking a ball. Rugby asks for passing it to a buddy… coherence 3, responds 2 | football coherence 5, responds 3 |
Common Misconceptions
- Masking is not training. It removes forbidden tokens but does not teach wording.
- Large literary text is not automatically useful. A 50,000 words e-free book did not beat the masked baseline.
- A higher usable rate is not perfect quality. At 60.5%, some replies become short and off topic.
- RL did not solve character awareness. It learned to satisfy the judge under a mask.
Action Advice for LLM Builders
- Start with a hard metric: 200 replies, 50 prompts, 4 answers each, and require both semantic scores to reach at least a 4.
- Use masking for strict lexical constraints, but follow it with SFT on task-shaped examples, not generic e-free prose.
- Use RL when you can score the behavior automatically or with a judge; here, judge scoring raised the SFT adapter to 60.5%.
- Expect reward collapse: monitor 300-token cap, short answers, and off-topic substitutes such as clik for cliché.
- Plan for small GPUs: the LoRA training loop ran on a 16 GB card, but memory still mattered at 10 to 13 GB.
FAQ
Why did Qwen3-1.7B fail with simple instruction?
Because tokens do not map cleanly to letters. The model cannot directly inspect whether a candidate token contains e, even if the output looks like words to a human.
Did the mask solve the task?
No. It blocked e and é, but it produced nonsense such as repetitive strings. Usable rate stayed at 26% without training.
Why did SFT help?
It provided prompt-shaped examples. The 180-example run reached 35%, and doubling from 90 to 180 added only 3.5 points.
Why did RL help more?
The judge optimized coherence and prompt response, not only e absence. The SFT adapter with RL reached 60.5%, while the masked start reached 43%.
Which auxiliary numbers are kept from the report’s numeric trail?
The retained anchors include 18, 24, 40, 42, 52, 53, 70, and 92.
For LLM practitioners, the value of this experiment is not the no-e trick. It is the measurable gap between token blocking, imitation learning, and preference-style RL. The model can reach 60.5% usable replies without ever becoming a character engine, and that boundary tells engineers where the real weakness lies.