Industry
June birthdays: organized lists and source notes
June birthdays: the pipeline an entry travels through stage by stage, why precision is data in its own right, and why cleaning a record is really editing.
Most pages of people born in a given month are not written. They are queried. A structured source is asked for everyone whose birth month equals six, the results are tidied, and a page comes out. That process can be done well. It is usually done invisibly, and the invisibility is the problem.
What to take away
- A generated list is a query, a snapshot, and a set of defaults. All three should be stated on the page, and almost never are.
- Precision belongs to each date individually. A source that stores it lets you keep it. A page that drops it turns years into days.
- The cleaning step is where the compiler's judgment enters. It should be visible, not buried in code.
The pipeline, stage by stage
| Stage | What happens | Where it goes wrong |
|---|---|---|
| Query | Ask the source for people whose birth month is June | The filter runs on a field that does not mean what the page will claim it means |
| Filter | Keep those who meet an inclusion rule | The rule is whatever was easy to express in the query language |
| Join | Attach names, fields of activity, links | Two records for one person, or one record for two |
| Clean | Fix formats, drop duplicates, settle conflicts | Every fix applied silently |
| Render | Produce the page | Display collapses whatever precision the data had |
| Snapshot | The page reflects the source at one moment | The moment is not recorded and the source moves on |
Precision is data too
A structured source such as Wikidata stores a date together with its precision: this value is known to the day, that one to the month, another to the year or the decade. It can also record which calendar the date is in, a distinction the March page explains, and it can tell apart a value that is unknown from one that is known not to exist.
A rendering step that ignores precision does something specific and bad. A year-only value still has to be displayed, and the lazy display is the first of the month or the first of the year. The person then appears on a day page they have no connection to, and a June list picks up people known only to have been born some time that year. The error is silent because the output looks exactly like a correct entry. The January index requires a date type on every entry; a precision flag is the same discipline one level down.
Cleaning is editing
Data cleansing sounds mechanical. For a people index it is a chain of judgment calls. Which of two conflicting dates to keep. Whether two similar records describe one person. Whether a value outside the expected range is an error or a real outlier. Each is an editorial decision, and a pipeline that makes them in code makes them without leaving a trace.
The fix is not to stop cleaning. It is to log it. Each row that was changed, merged or dropped should carry a note of what was done and why, so that when a reader challenges an entry the answer is a lookup rather than an excavation.
Declaring what the page is
A generated list can be trusted when it says what it is. The declaration has five parts.
- The source queried, by name and by version or date.
- The query in plain words: which field, which condition.
- The inclusion rule applied after the query.
- The date of the snapshot.
- What was done to the results by hand.
With those, a reader can reproduce the page and judge how far to rely on it. Without them, the page is a claim that some database once said something. The April page shows one way the queried field and the page's meaning drift apart; the May page shows why a generated count means less than it appears to.
Common questions
Is a generated list worse than a hand-compiled one?
It is different. It is more complete and more uniform, and its errors are systematic rather than scattered. A systematic error is easier to find once and easier to publish a thousand times.
Why not exclude every date with less than day precision?
That is a reasonable rule, and it should be stated. It also removes the people whose records are oldest or thinnest, which changes what the page covers without changing how it looks.
How often should the snapshot be refreshed?
Whenever the source has changed in a way that affects the page, which you can only know if you recorded what the last snapshot contained. Refreshing without that record swaps one unknown state for another. A year-only value is also the safest thing to hold for a living person, for the reasons the February page gives.