Introduction
Arguments for and against the applicability of machine learning in the production of intelligence products for military missions are dominated by technical, legal, ethical and organisational discourses. A fifth obstacle that trumps them all goes underscrutinised. US and allied military use of machine learning in intelligence products enjoins the military, the state and its citizens in a sprawling public–private sector digital ecosystem, including the digital data supply chains relied on for model training. These supply chains include nominally deidentified data collected via the mass surveillance of US citizens’ everyday activities by private companies. In turn, companies establish behavioural baselines derived from these neutral activities, from which a variety of derivatives can be produced, including behavioural anomaly detection for lethal military target identification. This manifests an overlooked institutional fault line, namely, the militarisation of the everyday life of the citizen. In turn, when military targeting collateralises the death of innocents, it enjoins the citizen in slaughter. This essay argues nothing short of the normativity of state-sanctioned killing on citizens’ behalf is at stake.
New tech, new enterprise and a new institution
Contributing to a 1999 RAND monograph on the American military enterprise in the digital information age, Carl H. Builder identifies a point of tension in civil–military relations which has deepened since that publication and remains unresolved. His central point was as follows: the transformative nature of the digital information age means the US military enterprise will not simply be applying new tools and methods to existing roles and missions. Rather, it will become a new enterprise with new roles and missions. It follows that the new enterprise will enjoin a new relationship with the society from which its resources and mandate are drawn. The recent development by private companies of machine learning models trained on large data sets made possible by mass surveillance of everyday civilian interactions means we can now begin to make sense of that new relationship and assess its implications for the institutional life of the military, society and the state.
The normativity of killing
As John Keegan (1934–2012) writes in his 2011 A History of Warfare, warrior culture follows society, but at a distance. Trends in economics and society make their way in and out of military affairs, but the transferral is never total, nor are the effects uniformly distributed across sectors. Disrupting business models and social norms in entertainment, commerce, social life and epistemology is categorically distinct from disrupting the business of killing. Under normal modern political conditions, the military enterprise remains institutionally alienated from society, because its primary purposeful activity is the sanctioned killing of human beings. At least since Thomas Hobbes (1588–1679), the modern state has been the institutional custodian of these normative relations.
Societies from which the military’s resources and mandate are drawn thus have a normative stake in its roles and missions. In the US, society’s stake is manifest formally via the legal restrictions it imposes on the military and the political control of its use in peace and war by executive government, as well as congressional control over budgets. The normative stake is manifest informally. At an intimate level, the families, loved ones and communities from which military personnel are drawn and put in harm’s way care about the meaning of what they do and why they do it. Society at large is like this but scaled up. In a democratic, rule-of-law society the values and norms attached to its unique form of political community are often flagged as what makes the act of killing its enemies justifiable and legitimate.
Likewise, militaries seek normative grounding for their roles and missions in the character of the society they swear to defend. Anecdotally, the serving members of no other institution express greater normative investment in the flag on the shoulder of their combat fatigues and the declared values it represents. Rule of law, the inalienable rights of the individual, freedom from coercive oppression, the democratic right to speak truth to power – these are meaningful connections to the character of political community felt by the military personnel from the US and Australia I have met and interacted with. The professional and cultural sanction against the targeting of civilians is another prominent feature of military cultural identity.
In sum, the normativity of killing as a profession is deeply woven into the institutions of the military and society. Discourse on technical, legal, ethical and organisational issues presented by military use of machine learning in intelligence products surfaces this normativity without capturing its profound implications for civilian–military relations and the state. Matters of the how and why of sanctioned killing go beyond these discourses, which notoriously tend to obscure more than they clarify, particularly in matters of ethics and technology. Which brings us to the normative implications of at-scale machine learning.
Machine learning and military judgement
The state has eschewed its custodial responsibilities in this regard, preferring to cheerlead based on a host of shaky assumptions. The related role of the large consulting firms in the neoliberal era is receiving renewed attention in turn. While scholars such as Linda Weiss and Mariana Mazzucato have shown the US government’s role in cultivating digital technologies has not been passive, the disruptive power of digital technology has meant the capacity to control its trajectory has been highly protean. The commercial machine learning industry, perhaps for obvious parochial reasons, has preferred not to address the hard question of institutional unravelling, favouring instead those discourses that surface technological and legal puzzles which, while problematic, have yet to succeed in derailing industry practices, aims and investment streams.
The US military enterprise has found itself drawn along here to some degree. By way of explanation, Sean T. Lawson notes how influential the misappropriation of various insights imported across scientific disciplines has been on military discourse, effectively creating a muddle of tactical acumen and strategic aims. Builder notes the more mundane reality of service parochialism and funding imperatives. On the options available to the enterprise as it adapts to the digital age, he writes:
Whether the choice is real or not may be less pertinent than the fact that there are factions within the American military that are willing to make the choice seem real to those in and out of uniform who must decide how the military should be organized and funded.
– Sean T. Lawson
The imperative of sound military judgement penetrates arbitrary levels, and ultimately cannot tolerate for long a discourse based on what ‘seems real’. Machine learning application to intelligence production is often sold as the enhancement of decision-making. The military is daily implored by private-sector consultants and vendors to embrace a ‘data culture’ lest they be rendered the Luddites of the digital era. The allure of faster tactical-level decision-making is not, however, synonymous with better judgement, and when judgement is considered at the strategic level, the tables can turn. Data-driven strategic blindness looms for a military enterprise not alert to the hard case of institutional unravelling. Again, Builder noted as much over two decades ago, ‘The balancing act is how to embrace the information technologies without being institutionally undone by them.’
Digitally entangled with private citizens
Machine learning applied to intelligence production for the military involves the training of models from large data sets. The expectation is the model, after training, will ingest new data and provide enhanced insight into the target phenomena via probabilistic statistical inference. These insights can then be included in intelligence assessments if they are procured by the military user. Large data sets for training allow the models to be iteratively fine-tuned. The ‘tuning’ is merely the adjustment of a host of statistical parameters internal to the model via a process called ‘backpropagation’, whereby a known outcome is ‘fit’ to the training data being input. The model will unavoidably end up either ‘overfit’ or ‘overgeneralised’. In the former, the model tends to treat new data it hasn’t been trained on as extraneous. In the latter, the model tends to treat new data it hasn’t been trained on as intrinsic. Either way, models are problematic when they encounter nontraining data and need to be treated with scepticism. This is the ‘bias’ problem and the ‘black box’ problem wrapped up together. It is typically treated as a technical obstacle.
When the entire digital ecosystem is interrogated, however, the normativity problem comes into view. Mass surveillance of neutral human behaviour has ridden alongside the digital revolution under commercial arrangements. Without this commercial ushering in, large data sets for model training simply would not exist. Behind extensive discourses on the privacy, ethical and legal issues presented by the commercial turn of the digital age lurks the normative question of the citizen’s right to obscurity in free and open society. The right to obscurity had not been formalised into legal frameworks prior to the digital age because prior to the advent of ubiquitous, continuous mobile computing, it was the default condition of every citizen, whose neutral behaviour was of no commercial value. The state presided over the citizen’s default obscurity, under which specific circumstances had to be met for it to be violated. We have left this world behind.
Salome Viljoen has theorised ‘horizontal data relations’, addressing how the ‘datafication’ of everyday life at the individual level expresses effects which must be understood at the population level: ‘Individualist data subject rights cannot represent, let alone address, these population-level effects.’ The implications of this analysis for civilian–military relations have been underscrutinised, but reaching back two decades to Builder’s work provides the necessary perspective. For our purposes, let us state the problem clearly. The surveyed condition of everyday citizens is inextricably connected via horizontal data relations to the generation of statistical inference, which may lead to the end user of an intelligence product prosecuting its military mission. In other words, connecting citizens directly to killing and to its collateral implications.
Conclusion
Scholars such as Builder and Keegan, and many others anticipating the digital age, are worth revisiting for the institutional focus of their analysis. International relations and security studies scholars can feel bamboozled by technocentric discourse which tends to dominate the mainstream. This is a shame. Any sufficiently transformative technology regime will impact most consequentially at the institutional level, and when military affairs are enjoined, we are pressed to consider the normativity of killing as the foundation of the modern state and its mandate to govern free people. Further, when society asks of the military that which, in order to deliver, the latter must transform itself, what the military comes to ask of society will be commensurate. Technology which masquerades as a free pass is not that. Nowhere is this collision more urgently in need of better understanding than in the area of machine learning and its applicability in matters of military judgement.
