mhoye

We've got a bunch of research papers that have come out of Mozilla's willingness to share a lot of our data. Bugzilla, lots of other stuff.

We pay attention to that research. We read it carefully, and our operational decision-making is informed by those results.

But we live and work in an evolving world, so the other thing we do as a matter of course is re-check our data to confirm our assumptions.

What I _want_ to be able to do is to go back and say, we based this decision in part on the results of paper X. Is that result still valid?

In a sane world, that's 3 steps. - Clone the author's VCS - Run author's [whatever] against an up-to-date dataset. - Look at the new graph.

Does the data still support the thesis? Great! Full steam ahead! Has something changed? Pause and think! Either way we're winning.

But that never happens. Because CS researchers don't publish code or data. They publish LaTeX-templated Word docs as paywalled PDFs.

@mhoye What are some concrete steps that a large foundation could fund to help out here?

@StuartBuck1 There are a few approaches a prominent fund-granting org could take (modulo politics, partnership complexity, investment, etc)

@StuartBuck1 Simplest is to insist that open publication be a condition of a grant. US-NSF, Canadian gov't do this, but: what is open?

@StuartBuck1 If "open" is "a .pdf on http://arXiv.org ", that's... uninteresting. It's OK by current standards, but we can do better.

@StuartBuck1 Next steps: support for the practices/processes of openness; hosting (easy), training (harder) & institutional support (Hard).

@StuartBuck1 (Institutional support means a lot of things, perhaps obviously, but Twitter does not permit a ton of specificity.)

@StuartBuck1 My knee-jerk reaction is "containerize osf.io, host open pub results and offer @swcarpentry training to all your grantees", but

@mhoye Agreed but ¬ looking in the right places. See http://www.artifact-eval.org/ . Huge movement, big changes, short time. Still more happening.

@ShriramKMurthi We're getting there slowly, yeah; Open Access in general is making good progress, still a long way to go though.

@mhoye Artifact Eval orthogonal to OA. It's directly tackling software issues. But we are also making OA progress. I'm pushing hard.

@ShriramKMurthi A philosophical point, but FWIW I think artefact eval and OA are deeply related in CS, particularly WRT cost-to-reproduce.