The values encoded in ML research: thoughts by Prof. Tom Dietterich

Two days ago, Prof. Tom Dietterich (Distinguished Professor - Emeritus, Oregon State University), posted a very interesting tweet about the values encoded in Machine Learning research, described in https://bit.ly/3zqPKpk

I quote his original tweet:

"Thoughts upon reading https://arxiv.org/abs/2106.15590: In this paper, the authors compare highly-cited papers from 2008-2009 with papers from 2018-2019 published in NeurIPS and ICML and summarize the values (i.e., desirable aspects) highlighted in those papers. The values are exactly what you would expect of an engineering field: ML researchers want algorithms that work well, that are computationally efficient, that generalize beyond the training data, that are novel, and that apply to the real world. Methodologically, ML researchers value quantitative evidence, theoretical guarantees, sound methodology, unifying ideas, and scientific understanding. Virtually absent was concern about the end users of ML systems (interpretability, fairness, privacy, user control, bias, etc.) or the potential downstream harms of deployed ML systems. (I'm using "end user issues" as a shorthand for the whole range of downstream issues). I suspect that part of this is a consequence of the authors' decision to focus only on highly-cited papers. These tend to be papers that introduce a new broadly-applicable technique (network architectures, training procedures, tools). Most papers lack such broad applicability, and this includes papers with an end-user focus. Ironically, elsewhere in the paper, the authors criticize the use of expected performance metrics, because they ignore rare but important cases. Yet that is what the authors have done. The http://cs.CY (computers and society) category on arXiv, has grown rapidly. There has been an explosion of ML work on fairness, interpretability, explanation, and bias. It is just not reflected in the 100 top-cited papers. The authors' analysis identifies a lack of emphasis on learning with small data sets. This reflects the focus on massive data sets of the deep learning revolution. Papers in the 1990s plotted learning curves (performance as a function of data set size). Today, learning curves are absent or focus more on the number of model parameters rather than the amount of data. But if we look in the NLP & computer vision conferences, we see quite a lot of work on few-shot learning and low-resource languages, and these are widely recognized as important problems. So this finding also reflects a ICML/NeurIPS sampling bias.

The paper also quantifies the rise of corporate funding for ML research by counting author corporate affiliations. The growth of corporate research labs has transformed the field, and this is reflected in the numbers. The paper says "The influence of powerful players in ML research is consistent with field-wide value commitments that centralize power." This is a very weak argument, and the wording reveals that the authors know this. The tight connections between university and industry research is also consistent with the fact that so many corporate tools (pytorch, tensorflow) and so much corporate research has been open-sourced. This open source value was not highlighted in the paper, but it is a central value of the ML research community and a key factor in the world-wide democratization of machine learning. Note that the Pytorch paper (from Facebook) is one of the 100 papers in the sample. I ask the authors: if all of the corporate funding had been replaced by government funding, would that be better? Surely having multiple organizations involved acts to reduce concentration of power. (The history of advances in computer science in the US is one of continual flow back and forth between corporate and government funding of research. (See Figure I.1 in https://www.nap.edu/catalog/23393/continuing-innovation-in-information-technology-workshop-report).)

Despite my criticisms, I share the concerns of the authors regarding the structural conservatism of machine learning, the defects of expected performance measures, and most importantly the need for user-centered ML tools. What is the path forward to addressing these problems, and which researchers will do this? Every end-user product is the result of a supply chain. Compilers, debuggers, and languages provide the substrate that supports ML algorithms and systems. HCI researchers and UX engineers design the human-computer interaction. DevOps and MLOps manage the deployed systems. ML algorithm researchers are tempted to ignore end user issues and rely on HCI/UX folks to handle them. But end user problems can't be fixed if the underlying algorithms and architectures don't provide the needed hooks. For example, UX designers can't provide actionable explanations or contestable UI if the underlying algorithms don't provide appropriate explanations. Harms discovered in deployment can't be addressed without upstream tools supporting problem reformulation and refactoring. A key challenge is to propagate downstream requirements upstream to earlier stages and provide the research infrastructure for ML algorithm researchers to develop methods that meet those requirements. This requires teams that span multiple pipeline stages and who can work together to formulate the upstream requirements, design valid metrics, and create the databases, simulators, and research protocols for executing the research. Industrial labs--with their tighter connections to product teams--are in a good position to contribute to this process. Universities--with their broader coverage of social science and humanities--also have a central role to play.

My most fundamental concern is that the "AI" narrative places far too much emphasis on automation and far too little on empowering end users, organizations, and institutions to achieve their goals. The narrow, mathematical view of AGI is at best a sterile and useless form of AI and at worst could lead to great harms. Fortunately, many many groups are adopting a human-centered vision, which makes me optimistic for the future. Some resources: https://dl.acm.org/doi/abs/10.1145/3419764, https://onlinelibrary.wiley.com/doi/full/10.1002/hbe2.117, https://hai.stanford.edu/, https://hmi.anu.edu.au"