PostCraft: Turning an LLM Writing Pipeline Into Something I Could Actually Trust

PostCraft is a local editorial pipeline for turning software repositories into credible technical portfolio posts. The goal is simple to describe but surprisingly easy to get wrong: take a project on disk, inspect the actual files, understand what the code supports, and generate a technical article that explains the problem, the design decisions, the trade-offs, and the professional value of the work without inventing a nicer story than the repository can justify.

That last part matters. A portfolio post is only useful if it is credible. If an LLM turns a small CLI into a “production-ready scalable platform,” it is not helping. It is making the project look inflated. PostCraft is my attempt to build the opposite: a writing pipeline where facts come before narrative.

Core idea: the LLM can help with drafting, but the repository decides what the article is allowed to claim.

The current pipeline scans a repository, selects relevant evidence files, renders project facts, runs a repository analyst, proposes article angles, creates portfolio positioning, builds an outline, writes a draft, reviews the draft technically, and either produces a final local post or blocks finalization. Every run writes a workspace with intermediate artifacts such as file_inventory.md, selected_files.md, project_facts.md, technical_analysis.md, post_angles.md, portfolio_positioning.md, outline.md, draft.md, technical_review.md, final.md, and warnings.md.

That audit trail turned out to be essential. It gave me a way to inspect not only the final output, but also the chain of decisions that produced it.

Where the Pipeline Started

One of the first useful test cases was a Rust project named ha-client-rs, a Home Assistant CLI client. The scanner did something promising: it selected files that looked genuinely useful for understanding the project, including README.md, Cargo.toml, makefile, src/cli.rs, src/main.rs, src/app/prompts_service.rs, and src/infra/ha_repo.rs.

That evidence contained concrete behavior: command-line parsing, Rust dependencies such as clap, reqwest, and serde_yaml, Home Assistant interaction, configuration files, sensor-related operations, calendar events, and daily prompt generation.

But the generated draft did not use that evidence well.

The first draft.md was not really an article. It mixed an outline, technical analysis, and portfolio positioning in the same file. It contained headings such as ## Outline, Technical Analysis, and Portfolio Positioning. In other words, the writer agent was leaking pipeline artifacts into the post instead of transforming the plan into prose.

The technical reviewer correctly blocked finalization. It flagged generic claims such as users “often struggle” with Home Assistant workflows and pointed out that some statements were plausible but not directly supported by file evidence. That was a good sign: the reviewer was useful. But it also showed that the pipeline was still relying too much on the LLM to behave well by instruction alone.

The problem was not that the pipeline failed to run. It created a complete workspace. The problem was that “the pipeline runs” is a much weaker claim than “the generated article is good enough to represent my engineering work.”

Failure mode: the system was producing files, but the files were not yet reliable enough to publish.

The First Lesson: Artifact Existence Is Not Quality

The initial tests mostly verified plumbing. They checked that a workspace was created, that files existed, and that the finalization step could be blocked when a fake reviewer returned BLOCK.

That was necessary, but not sufficient. A malformed draft.md can exist. A vague technical_review.md can exist. A final.md can exist while still being unusable as a portfolio article.

So I added evals around the actual product risks:

• Is the draft a standalone article?
• Does the draft leak pipeline sections such as ## Outline or Technical Analysis?
• Does the draft contain enough prose paragraphs?
• Does it avoid generic marketing phrases such as scalable, production-ready, full-featured, or must-have?
• Does the technical review include a valid PASS or BLOCK verdict?
• Does the review tie claims back to file paths?
• Does the reviewer avoid classifying the same claim in multiple categories?

The first self-eval run was intentionally uncomfortable. It failed on exactly the kind of issues that are easy to miss when reviewing generated text manually:

FAILED test_self_eval_draft_is_standalone_article
FAILED test_self_eval_draft_has_no_forbidden_headings
FAILED test_self_eval_draft_has_prose
FAILED test_self_eval_draft_avoids_generic_phrases
FAILED test_self_eval_review_references_file_paths

That was the turning point. The evals stopped being abstract test coverage and became a feedback system for the product.

I ended up organizing the evaluation system into four layers: artifact contracts, claim safety, scanner evidence selection, and agent orchestration.

Layer 1: Artifact Contract Evals

The first layer was deterministic. It did not need an LLM. It inspected Markdown artifacts and checked whether they respected basic contracts.

For draft.md, the rules were deliberately simple:

• it must start with an H1 title;
• it must contain prose paragraphs;
• it must not contain pipeline artifact headings;
• it must not look like an outline dump.

For technical_review.md, the rules checked for a machine-readable verdict and consistency between review verdict and warnings.

This layer is not trying to judge literary quality. It is a guardrail against obvious failures. If the writer returns an outline instead of an article, no amount of later review should pretend that the result is ready.

Layer 2: Grounding and Claim Safety

The second layer focused on claims. I added helpers to detect generic phrases and unsupported architecture labels. For example, terms like production-ready, scalable, full-featured, and must-have are not banned forever, but they should not appear unless the repository provides concrete evidence.

A concrete case would be a claim like “a production-ready Home Assistant automation platform”, which should be rejected unless the repository contained evidence for deployment, operational guarantees, or production use. A narrower claim such as “a CLI that wraps Home Assistant interactions for configuration, sensors, calendar events, and prompt generation” is much easier to support from the selected files.

I also added checks for technical reviews. A review is more useful when it references file paths and keeps claim categories mutually exclusive. A claim should not be both “supported” and “exaggerated” unless the reviewer is explicitly distinguishing two different versions of the claim.

This surfaced an interesting implementation detail: parsing LLM-generated review text is messy. The review might put the claim in a bullet and the evidence in a continuation line. The parser had to learn to treat lines such as Evidence: and File path: as part of the same claim block, otherwise the eval would miss evidence that was actually present.

That is a small detail, but it reflects a broader lesson: evals for LLM workflows often need to evaluate semi-structured text, not perfect JSON.

Layer 3: Scanner and Evidence Selection Evals

Once the artifact-level checks existed, the next problem was upstream. If the scanner does not preserve enough factual evidence, the writer and reviewer are forced to infer too much.

I added fixture repositories under tests/fixtures/repos/:

• a Rust Home Assistant CLI fixture;
• a Python layered CLI fixture;
• a minimal single-file script.

The Rust fixture mirrors the shape of the ha-client-rs project: Cargo.toml, README.md, makefile, src/main.rs, src/cli.rs, src/app/prompts_service.rs, and src/infra/ha_repo.rs. The evals check that the scanner detects Rust, selects balanced evidence, preserves enough excerpt content to identify behavior, detects automation and entrypoints, and does not exceed the selected-file cap.

The Python fixture checks a different shape: pyproject.toml, README.md, Makefile, a CLI entrypoint, and src/domain, src/application, and src/adapters modules. This helps verify that architecture-related signals are not hardcoded to one language.

The minimal script fixture is just as important. It protects against overclaiming. A single script should not trigger architecture or documentation signals just because the system wants every project to sound impressive.

One Rust dependency eval is currently marked as expected failure because Cargo.toml dependency parsing is not implemented yet. I made that xfail strict so it becomes visible when the scanner finally supports Cargo dependency extraction. That turns a known gap into an explicit contract.

Layer 4: Agent Contract Evals

After scanner evals, I added agent contract tests with fake LLMs. The goal was not to test model quality, but to test orchestration:

• the writer receives outline, technical analysis, and portfolio positioning;
• the reviewer receives draft, project facts, and technical analysis;
• the architecture narrator receives facts and analysis;
• the outline agent receives post angles and positioning;
• a BLOCK verdict prevents finalization;
• a PASS verdict allows the final editor to produce the final post;
• warnings reflect review state;
• malformed writer output is repaired before the reviewer sees it.

This required a better fake LLM. A simple fake that returns the same response to every prompt is not enough for a multi-agent pipeline. It can make tests pass or fail for the wrong reason. I replaced that with a scripted fake that returns different responses depending on the prompt: one response for outline generation, another for writing, another for repair, another for review, and another for final editing.

That let me test a real contract: if the writer returns a malformed draft with ## Outline and Technical Analysis, the pipeline should repair it before sending it to the technical reviewer. The eval now confirms that the reviewer receives the repaired draft, not the outline dump.

What Changed in the Pipeline

The evals led directly to implementation changes.

The writer prompt became stricter. It now tells the model to use the outline as a private writing plan, not as content to paste. It explicitly forbids sections such as Outline, Technical Analysis, Portfolio Positioning, Supported Claims, and Claims To Avoid in the final draft.

I also added a deterministic draft contract validator after the writer. It checks for an H1 title, forbidden pipeline headings, enough prose, and generic phrases. If the draft fails that contract, the pipeline performs a repair pass before technical review.

The technical reviewer prompt was tightened too. It now asks for file-path evidence when available. I also added conservative contract repair for cases where the LLM response is truncated before producing a parseable verdict. If a review contains unsupported or weak claims but no final verdict, the pipeline can conservatively append a BLOCK verdict rather than silently treating the review as usable.

These changes moved PostCraft from “trust the prompt” toward “trust the contract.”

What the Evals Achieved

The most important improvement is not that the tests pass. It is that the system now has a clearer definition of failure.

Before the evals, a bad run required manual inspection. I had to open draft.md, notice that it was an outline, read technical_review.md, and infer where the pipeline went wrong.

After the evals, the failures became explicit:

• draft.md must be a standalone article;
• draft.md contains forbidden headings;
• draft.md contains generic phrases;
• technical_review.md references no file paths;
• duplicate claims appear across reviewer sections.

That changed the development loop. Instead of asking “does this output feel better?”, I can ask “which contract does this output violate?”

The deterministic eval suite now covers artifact contracts, grounding rules, scanner evidence selection, and agent contracts. A recent deterministic run across those layers passed with known expected failures still marking future work. The LLM-backed self-eval sits outside those deterministic layers: it is not the contract itself, but a broader smoke test against a real project run. It remains useful because it runs the pipeline against the PostCraft repository itself and catches behavior that only appears in a real generation.

The latest self-eval still exposed a remaining reviewer-consistency issue: the same src/shared/ path appeared across supported and weak claim categories. I do not see that as a failure of the eval strategy. I see it as proof that the evals are now sensitive enough to find the next problem.

Engineering Takeaways

The main engineering lesson is that LLM pipelines need product-specific evals, not just generic tests.

Unit tests can tell me that a function returns a string. Integration tests can tell me that a workspace was created. But neither tells me whether a generated portfolio article is grounded, readable, and safe to publish as a representation of my work.

For this project, quality means several concrete things:

• the article must be standalone prose;
• the scanner must preserve useful evidence;
• the writer must not paste planning artifacts;
• generic claims must be treated as suspect;
• the reviewer must cite file evidence;
• finalization must block when unsupported claims remain;
• expected gaps must be tracked as strict xfails.

Those are not abstract LLM evaluation principles. They come directly from the product I am building.

Why This Matters for the Portfolio

PostCraft is itself a portfolio project about building portfolio posts. That makes quality especially important. If the tool exaggerates, it undermines its own purpose.

Adding evals forced me to encode the editorial standards I actually care about: technical grounding, traceability, cautious claims, and clear separation between evidence and narrative. It also made the project more interesting technically. The work is no longer just “call an LLM and save Markdown.” It is a multi-agent workflow with auditability, deterministic contracts, fixture-based scanner evals, scripted fake LLMs, and self-evaluation against the project itself.

That is the kind of engineering signal I want the project to communicate: not that the first generated draft was perfect, but that the system was designed to reveal its own weaknesses and improve from them.

Next Steps

There are still clear gaps.

The scanner should extract dependencies from Cargo.toml, not only from Python manifests. The outline phase should reject generic problem claims before they reach the writer. The reviewer needs stronger validation to prevent the same claim from appearing across supported, weak, and exaggerated categories.

But the direction is now much healthier. PostCraft has moved from a pipeline that generated artifacts to a pipeline that can be evaluated against the quality bar it claims to enforce.

That is the real progress: not just generating text, but building a system that can tell when the text is not good enough.

This article was written with the help of artificial intelligence, as you might expect.