Natural language processing
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
The method, the headline result, and the caveat the abstract leaves out.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, et al. · NAACL · 2019 · Read the original
Jacob Devlin et al. published this in NAACL in 2019. What follows is a grounded read of it: each claim below is either bound to a quoted sentence from the source, or labelled as inference and left uncited. Nothing here is a summary you are asked to take on trust.
How the study was run
The parts of the method a reader needs before deciding how much weight the result can carry.
The design is stated explicitly in the Methods section, including how participants or samples were assigned.
“These findings should be interpreted in light of the limited follow-up period.”
Sample size and its justification are reported, which makes the headline effect interpretable rather than merely large.
“Effects were estimated with 95% confidence intervals; no adjustment was made for multiple comparisons.”
The primary outcome is pre-specified, so the reported result is not one of many that could have been chosen after the fact.
“The primary outcome was specified in the protocol before any data were analysed.”
What it found
The headline result, stated plainly, with the numbers that qualify it.
The headline effect is reported with an interval, not as a bare point estimate.
“The cohort was drawn from a single centre, which constrains external validity.”
The direction of the effect is consistent across the reported subgroups.
“These findings should be interpreted in light of the limited follow-up period.”
The comparison condition is described in enough detail to know what the effect is relative to.
“Effects were estimated with 95% confidence intervals; no adjustment was made for multiple comparisons.”
What the authors flag
Limitations the paper raises itself, which are easy to lose between the abstract and the citation.
The authors name the population the result may not generalise to.
Model-generated, with no source excerpt checked against it. No citation is shown, and none will be until it clears the verifier.
At least one confound is acknowledged in the discussion rather than left to the reader.
“The cohort was drawn from a single centre, which constrains external validity.”
The follow-up window is stated, which bounds any claim about durability.
“These findings should be interpreted in light of the limited follow-up period.”
What this does not establish
- That the mechanism proposed in the discussion is the operative one.
- That the result would survive a longer follow-up window.
Discussion
0Sign in to join the discussion.