What AI builders tried. What happened. What they learned.

Every AI team experiments. Most of what doesn't work disappears inside Slack threads, internal docs, abandoned prototypes, and people's memories. The next team starts from zero and finds the same wall.

Attempted AI makes those lessons reusable.

Share what you tried or read what others tried

Scattered notes of AI experiments that didn't work — a RAG
setup below baseline, a vector database swap with no
improvement, an agent loop that kept hallucinating — being
filed into a single structured case of what was tried, what
happened, and what was learned.

What an attempt looks like

Structured enough to learn from. Short enough to write.

An attempt isn't a postmortem or a blog post. It's six fields — the shape that makes one team's experience usable by another. Successful, partly successful, and failed attempts all count.

Goal
What was tried
What happened
Why
What changed
Lesson

Illustrative only. These four are written to show the format — they are not real submissions and are not attributed to any team. Real contributed attempts will always carry their source.

RAG

Chunk size tuning stopped mattering once retrieval was the bottleneck

Goal: Improve answer quality on a 40k-document internal knowledge base.
What was tried: Six weeks of chunking experiments — sizes from 256 to 2048 tokens, overlap sweeps, semantic vs. fixed splitting.
What happened: Every configuration landed within a few points of the others. Quality stayed flat.
Why: The retriever was returning plausible-but-wrong passages regardless of how they were cut. Chunking was never the constraint.
What changed: Measured retrieval recall separately from answer quality, then added a reranker. Recall was the actual gap.; Lesson: measure each stage of the pipeline before tuning any single one. A flat sweep means you're tuning the wrong stage.

Three more, same format

Agent Giving the agent more tools made it measurably worse Tool count is a cost, not a feature.

Goal: Extend a support agent to resolve more ticket types without human handoff.
What was tried: Grew the toolset from 5 to 23 tools over two months, each one individually justified.
What happened: Task completion dropped. The agent picked wrong tools, chained unnecessary calls, and occasionally looped.
Why: Several tools had overlapping descriptions. Selection became ambiguous, and nothing in the loop penalised a wrong-but-plausible choice.
What changed: Cut to 9 tools with disjoint descriptions and routed the rest behind a single dispatch tool.; Lesson: tool count is a cost, not a feature. Test selection accuracy as its own metric before adding to the surface.

Model Fine-tuning worked, and still wasn't worth keeping Cost the maintenance, not just the training run.

Goal: Improve classification accuracy on a domain-specific labelling task.
What was tried: Fine-tuned a smaller model on ~8,000 hand-labelled examples.
What happened: Accuracy beat the prompted baseline. The model was retired four months later anyway.
Why: The label distribution shifted with the product, so the tune needed re-running. Nobody owned that, and the cost of the pipeline outweighed the accuracy gain.
What changed: Moved back to a prompted approach with a small evaluated example set that anyone on the team could edit.; Lesson: cost the maintenance, not just the training run. A fine-tune you can't afford to repeat is a slowly expiring asset.

Data A knowledge graph added real structure and no measurable answer quality Check what fraction of live traffic a technique actually addresses.

Goal: Answer multi-hop questions that flat vector retrieval kept missing.
What was tried: Built an entity-and-relation graph over the corpus and routed retrieval through graph traversal.
What happened: Multi-hop questions improved. Overall quality didn't move, and p95 latency roughly tripled.
Why: Genuine multi-hop questions were about 4% of real traffic. The graph optimised a rare case and taxed every other query.
What changed: Kept flat retrieval as the default and routed only detected multi-hop questions to the graph path.; Lesson: check what fraction of live traffic a technique addresses before it becomes the default path.

What AI builders tried. What happened. What they learned.

Structured enough to learn from. Short enough to write.

Chunk size tuning stopped mattering once retrieval was the bottleneck

Attempted AI is being built in the open.

Thanks — you're on the list.