Building PublicFeed: A Data Pipeline for Government Updates

I wanted a simpler way to follow government updates. I built PublicFeed to collect official documents from India and turn them into readable summaries, with links back to the originals.

From a data engineering perspective, the useful part of this project is everything that makes those summaries repeatable and traceable. Sources go offline. PDFs contain scanned pages. A task can fail halfway through a batch. A better prompt can change what the model extracts from the same document.

Each of those cases needs a clear answer: what do I save, what do I retry, what do I recompute, and what should reach the website? This post walks through the pipeline I built around those questions.

Start with the data flow

PublicFeed uses Python for collection and processing, Apache Airflow for orchestration, and Flask for the website. The pipeline saves its evidence and intermediate results in files on persistent storage. The website reads a published JSON feed.

The main document flow looks like this:

flowchart TB
    Sources["Official websites and documents"] --> Raw[("Saved source files")]
    Raw --> Transform["Extract text, create a summary,<br/>and check the result"]
    Transform -->|"Passing records"| Feed[("Published JSON feed")]
    Feed --> Site["Flask website"]

The AI call is one transformation inside this flow. Airflow owns task execution, retries, and concurrency. Stored outputs connect the steps. The website does not call a model when someone opens a story.

That separation lets me change how documents are processed without making readers wait for the pipeline.

Collect incrementally, with a boundary around each source

I started with orders from the Jammu and Kashmir General Administration Department. One source was enough to build the path from discovery to publication before expanding coverage.

Each source has code that understands its listing pages, document links, and available metadata. The rest of the pipeline works with a shared record structure. This keeps a change to one department's website from spreading through the processing code.

Routine runs look for recent items and skip work already marked as finished. Targeted runs can select particular sources, widen the lookback window, or request reprocessing. Backfill limits and offsets let me work through a larger set in smaller batches.

There is an important limit here: a recent polling window is not a complete historical archive. Recovering a missed period requires choosing a window that includes it. Likewise, skipping a known item is efficient, but does not by itself prove that the remote document has never changed.

These are collection policies I need to understand when judging coverage, not just settings for making jobs faster.

Preserve raw data and keep derived results separate

Before extracting text or calling a model, the pipeline saves the source file. It identifies those bytes with a SHA-256 hash, so the same content can be reused without storing another copy.

The evidence store keeps several different records:

This gives me data lineage: I can trace a published summary back through its processing results to the document it came from. A correction can start from the saved evidence, rather than depending on the source website still serving the same file.

Extraction deserves its own step. A PDF with selectable text and a scanned PDF need different handling. Some reading modes also have page limits. If the input was only partly read, later checks cannot establish that the whole document was covered.

Make retries reuse work and reprocessing create a new version

A task retry and a change to processing logic are different operations.

If a model request fails after other items have finished, I want the retry to reuse the completed work. If I change the extraction logic or prompt, I want a new result that I can compare with the earlier one.

Derived results are therefore keyed by the input hash and the relevant processing versions. Depending on the step, those versions include the extractor, model, prompt, and source metadata. Analysis also depends on the extraction version, so changing how a document is read can invalidate the analysis that follows.

flowchart TB
    Key["Input hash + processing version"] --> Exists{"Saved result?"}
    Exists -->|"Yes"| Reuse["Reuse it"]
    Exists -->|"No"| Run["Run the transformation"]
    Run --> Save["Save a new result"]
    Reuse --> Next["Continue processing"]
    Save --> Next

Existing derived results are returned unchanged. A new version gets a new file. Repeated processing can therefore reuse the same saved output, even though a fresh model call might produce different wording.

The run manifest records progress after each item. Later processing steps read that state, and retries can pick up failed work without starting every transformation again. This makes recovery cheaper and gives me a record of why two runs produced different results.

Versioning still requires discipline: a behavior change needs the appropriate version bump, or an old cached result may remain eligible for reuse.

Isolate failures in Airflow

Each document source gets its own Airflow branch: collect, extract, analyze, and judge. The shared publishing steps run after all source branches have finished, including branches that failed.

The first shared task uses Airflow's all_done trigger rule. That allows successful sources to contribute results even when another website is unavailable. Later publishing tasks still depend on the preceding publishing task succeeding.

There is also a boundary within each processing task. An error on one document is recorded while the task continues through the other records. If every attempted record errors, the task fails so Airflow can retry it and show the failure.

This means a green task is not enough to judge data quality or completeness. I also need the record counts and failure reasons in its manifest.

Only one DAG run is active at a time. A polling sensor controls how often collection starts, and an Airflow pool limits concurrent model work. These controls keep overlapping jobs and expensive transformations manageable on the current deployment.

Treat model output as data that needs validation

The model produces structured details as well as a headline and summary: dates, actions, references, and supporting quotes. Those fields need checks before they become public data.

Code checks what can be tested directly. A quoted passage should appear in the extracted text. A claimed effective date needs supporting wording. Checks also look for certain personal details that should be omitted.

A separate AI review then compares the draft with the available text. A positive review cannot override a failed code check. Records that fail the publication checks stay out of the feed, while their saved processing results remain available for investigation.

Dates show why this matters. An order's issue date is not necessarily its effective date. “Within ten days of receipt” should remain a relative deadline; the download timestamp is not a valid substitute for receipt.

These checks have limits. Extracted text can be wrong, and the reviewer may use the same model as the writer. PublicFeed's summaries are AI-generated and have not been individually reviewed by a person. Keeping the original source attached to every story makes those limitations easier to investigate.

Model the document's time separately from the pipeline's time

I also wanted to preserve changes in what the system knows. Consider an illustrative example:

• An order says a rule takes effect on 1 November.
• The pipeline first observes the document on 5 November.
• A new analysis corrects an extracted claim on 6 November.

These dates answer different questions. The first describes when the rule applies. The second records collection. The third records a change in the system's interpretation.

The current fact history uses an append-only JSON Lines file. It records claims and closes earlier versions when their contents change. It keeps the document's stated validity period separate from the period during which a version was recorded, and also stores the observation time. This is the basis of the bitemporal model: when a claim applies and when the system knew that version of it.

An unchanged claim does not need a new version. An unknown effective date stays unknown. These are still candidate claims extracted by the pipeline, so the history can contain mistakes; saving them makes the changes traceable.

Publish a stable output for the website

Publishing is an incremental update. Passing items are added or refreshed by item ID. Items rechecked and found unsuitable are removed. Older items that were not revisited stay in the feed.

This is useful when a source fails: absence from the current run does not automatically delete its previously published stories.

The publisher writes the complete replacement feed to a temporary file, then renames it into place. Readers get the old or new file rather than a partially written update. A feed snapshot is also saved with the run.

For the current deployment, a JSON feed is a simple interface between the pipeline and Flask. The website can search, filter, and render already prepared records. It does not need access to the model service or all the intermediate evidence.

Operate the pipeline with coverage in mind

I keep development and production separate. Local runs use Ollama and local storage. Production runs its own collection and processing jobs with hosted models. A production snapshot can be copied locally to investigate a problem without making the live feed depend on my machine.

Source connectivity has needed attention too. Some government sites were reachable locally but dropped connections from the hosting region. A restricted proxy in India provides a working route for those requests while keeping HTTPS encryption intact.

The next improvements I want are clearer source freshness reporting, better visibility into partial collection, and more evaluation examples for difficult documents. Job status, source coverage, and summary quality each need their own evidence.

Building PublicFeed has been a practical exercise in ingestion, lineage, versioning, orchestration, and data quality. The summary is what the reader sees. The pipeline is what lets me explain where it came from, recover when processing fails, and improve it without losing the history.

You can explore PublicFeed or read how the summaries are produced.