Putting ‘qualitative versus quantitative’ in its rightful place

The distinction between ‘qualitative’ and ‘quantitative’ data/methods irritates me. It is often presumed that the latter is indicative of – at least in principle – greater scientific or analytical rigour, which is not necessarily wrong in some cases, but is only pertinent to one particular methodology of (social) science, where others can be rigorous as well despite no quantification taking place.

I will now half-assedly defend this position.

By rigour, I mean the extent to which an analysis is coherent, complete, and precise. The highest level of rigour would be that of a deductively valid proof – i.e. formal, incl. mathematical.

1. The Speciousness of Qual vs Quant

Pragmatically speaking, ‘qual’ and ‘quant’ do generally help us differentiate between analyses or data that are represented or studied according to substantially different degrees of formalisation. But a problem arises when we start to think that this distinction is coherent at a deeper philosophical level. It isn’t. There isn’t actually a difference, at least not essentially, between these two supposedly different types of measurements or analyses. Qualitative data speaks to the qualities of some phenomena or property, and qualitative analysis is the study or explication of those qualities. But quantitative data is the same thing, only expressed mathematically. There’s no deep methodological difference between saying something is ‘big’ or ‘small’ or whathaveyou and describing it according to some unit of measurement because units of measurement are themselves simply conventions against which other things can be compared; to employ them is simply to make use of a more refined set of qualitative indicators.

I see fit to mention this because the presence or absence of maths should not be the criterion of demarcation between different types of data or analysis. Sometimes ‘quantify that!’ is a reasonable request for more rigour, but often it isn’t. In many cases the reason why a scholar has not quantified their data and analyses is not because they are maths-phobic or because their observations and concepts are too ill-defined to be amenable to quantification, but rather because they are employing a (social) scientific methodology in which it is not useful or possible to translate their information into a mathematical language; the questions they seek to answer and the ontological/epistemological/metaphysical wagers upon which  their methodology rests limit the usefulness of mathematical formalisation.

This brings me to my next point.

2. The Neopositivist Chauvinism of Qual and Quant

Within one quite popular methodology of science, there is excellent reason to view quantitative analyses as, generally speaking, holding the possibility of more rigour. This is the methodology that has given us the power to predict and alter the natural world in startling and amazing ways: Positivism (and its descendents). Within the Positivist tradition, scientific explanation typically consists of law-like statements which take the following form:

If X, then Y follows.

For example, in my field one candidate law might be ‘if two countries are democratic, they will not go to war against one another’. In another field, it might be something like ‘the introduction of chemical X at time T1 will lead to a reaction R at time T2′.

These laws are tested by hypothetico-deduction, usually: the consequences that would obtain if the law were true are predicted in the form of a hypothesis, then tested by way of observation and experimentation. Sometimes the law itself is not discussed; a scientist may feel that it suffices to show that R did indeed occur, and allow the scientific community to infer what it will from this result. But explanation and causality within this methodology nevertheless reduces to laws of nature.*

This methodology pressures one into using increasingly larger samples, increasingly precise measurement (in order to test how much an increase in X leads to an increase in Y), and increasingly complex methods of analysis to test whether, across a wide range of cases, Y indeed does follow X when all other factors are controlled. Since the type of reasoning is inductive, an increase in the number of cases in which the temporal (read: causal) relationship between X and Y holds will lead to an increase of confidence in the truth of the law. It is easy to see how this will lead to the scientist wanting to use statistical techniques and formal analyses to determine with maximum confidence that it is not some intervening third factor or ‘fluke’ that is responsible for the apparent ‘X then Y’ relation (ie significance tests), and therefore that greater rigour is made possible through ‘quantification’, whereby the qualities of the data are made amenable to mathematical analysis.

In the social sciences, for various reasons of terminological clarity, this methodology has been well-referred to as ‘Neopositivism’.

However, Neopositivism – that is, social science enquiry via the Positivist tradition – is not very good for answering a whole host of interesting and important empirical questions, nor is it without strong competition in enabling a coherent conception of causality in complex systems.

When we study social behaviour, we are often interested not so much in finding predictive laws – ‘if we see X, we know Johnny will do Y’ – but rather to specify the motives that led someone to act as they did – ‘because of his perception X, Johnny feel it appropriate to do Y’. This is the core of explanation in ‘interpretive social science’: the scientist explains social outcomes by clarifying and explicating the Reasons why an actor took a certain action or set of actions. While it might be possible to test competing explanations across cases via Neopositivist methods, it is the near-consensus of Interpretivist social scientists that a far better, more rigorous method involves ‘thick description’ or other ways of developing a rich and nuanced picture of the cultural conventions and personal narratives that serve as the context that make action meaningful; that make action something other than a reflexive twitch.

Meanwhile, when we study complex systems, we often find that prediction is impossible, and we’re thus moved to seek an account of causality that doesn’t require us to reduce phenomena to laws of nature. Social scientists, as students of complex systems, often make use of single-case analyses in which one particular configuration of factors, entities, or processes is examined for how it caused a given outcome. Causation cannot be reduced to any particular factor or set of actors; rather, all factors ‘came together’ exactly in such a way as to produce the effect. This kind of analysis enables the scientist to determine what is possible and how that possibility can be realised. The social scientist may attempt to specify certain ‘causal mechanisms’ that constitute cross-case regularities of causation, but these mechanisms are not laws; rather than are types of patterns or processes that connect cause and effect via their instantiation, independent of the observable phenomena they produce. In this methodology, quantification and mathematical analysis is less useful than methods such as ‘process tracing’, because apparent regularities in conjuctions/correlations of initial conditions and outcomes indicate nothing about the causal sequences that lead to those outcomes.*

The Punchline

As I have tried to show, there are other methodologies for engaging in social enquiry that can and are conducted rigorously and which do provide interesting scientific conclusions. They are not well-aided by quantification and yet they still feature explanations that are formally coherent, deductively valid, and meticulously grounded in empirical analyses.

*This highly simplistic summary  should not be considered adequate by anyone.

Culture and ‘science’

A couple people have requested of me that I make a blog post about this topic. I am not sure I am going to contribute anything novel to the discussion, but I’ll try to at least bring in some of the literature and philosophy of science that I think is relevant and which many people ignore or don’t know. I am probably not going to link every name or concept, though I’ll try to link to more info on some of the less clear references. One could also skip to the end if they were in a huge hurry, but I don’t recommend it…

The ‘topic’ is whether science is a culturally contingent construct; whether it is reasonable to talk about ‘science’ as something ‘Western’, and no more warranted than ‘other ways of knowing’, perhaps from the ‘East’ or wherever.

The prompt for discussing this topic is an essay by Natalie Reed, whom I hold in general esteem for a great many reasons, but who has to a large degree missed something important with this particular contribution.

The premise of her essay is that it is wrong for people to dismiss science as no more reliable or epistemologically sound than other ways of enquiring, other ways of interrogating nature for its secrets, other ways of investigating reality. The people who dismiss science in this way typically come from positions of political and social subordination, and they view ‘science’ as a kind of hegemonic and oppressive construct that serves to delegitimate the philosophies of other cultures. Yet, according to Natalie, this is both foolish and clueless:

The differences between the “West” and other cultures aren’t so fundamental as to speak to how we think, how we feel, how we know, how we process knowledge….Even if it were true that science as we understand it is simply a “Western” construct being inexactly applied to a more universal kind of thought, does that say anything about science being wrong? Dangerous? Harmful? Does it’s relative “Westerness” have anything whatsoever to do with the applicability of science, or it’s beneficial nature relative to human bias?

Let me first praise Natalie for some things that I think she has done better than I probably could. She has, with empathy, sympathy, and authenticity, captured why it is that so many people are inclined to view ‘science’ and the accompanying narrative of progress as coercive. She also hammers on what one friend of mine has called ‘noble savage worship’:

To say that science is “Western” is intensely condescending, dismissive and Euro-centric. It takes the same old colonial narrative of the “advanced”, civilized peoples of Europe, and the savage mystical primitives of Everywhere-Else, and repackages it in such a way as to be enjoyed within the halls of contemporary academia without any post-imperial guilt.

Bravo! I am genuinely refreshed and delighted to see this sentiment expressed by someone who is otherwise enormously sensitive to the position of marginalised people in our society. I am filled with respect for Natalie’s condemnation of this sort of hypocrisy, and also for her steadfast belief that principles of scepticism, self-criticism, and methodological rigour will do more for the cause of the oppressed than mysticism ever will.

Except now there are some philosophical problems I want to discuss. The above paragraph hints at how Natalie understands science, and she makes this understanding more clear later in her essay (and pardon the long quote):

Science is by definition non-cultural. It is not a part of a struggle between different cultural worldviews. In so far as a cultural worldview falls into a scientist’s interpretation of her data, she’s screwing up. She’s making the kind of human error science is structured to minimize as much as possible.

Science is not a “way of knowing”. It is a process. A process designed to minimize all of the different little biases, cognitive distortions, logical fallacies and errors of perception that define a cultural perspective, or subjective vantage point, or “way of knowing”. It’s streamlining a bunch of different principles that have been practiced by all human beings in all cultures for millenia to help us tell what’s really going on from what simply seems to be going on, to tell what is probably true from what we want to be true, to tell the important variables from the coincidences, to tell the actual causal relationships from things that just happen to come after other things.

And it wants to be wrong. It wants to make sure it can be shown to be wrong. It questions itself, it’s open to criticism, it values self-questioning, skepticism… the things I fear our progressive movements don’t value nearly enough. It’s wrong over and over and over again, and it KNOWS it will be wrong again. It acknowledges its margin of error.

Let me deconstruct this a bit.

Oh wait, before I do, let me just point out that Natalie reifies ‘science’ in a way that seems problematic here. I caution against ascribing any desires or values to Science. ‘Science’ is not a mind, and it has no will. As Natalie herself points out, it is something involving processes. Those processes are employed and acted out by people – scientists – and we should avoid discussing science in a way that may lead us to forget that science is something that scientists do, whether or not we believe that there is a single, universal method that all scientists should follow. And despite her protests to the contrary, science as Natalie conceives of it is precisely a way of knowing. It is very evident that Natalie believes science to be a process of generating knowledge about an ‘objective’ reality, and that process involves ways of minimising such things as ‘observer bias’ or ‘inaccuracy’.

There are certain assumptions to Natalie’s position. Let  me spell out the ones I think are salient:

1. We live in an ‘objective reality’ independent of our experience of it; there would be things like mountains and trees whether or not we were around to believe in them;
2. We can observe that ‘mind-independent’ reality;
3. Our observations are warped and flawed due to our cognitive biases, but those biases can be minimised to produce knowledge of reality that is more accurate or approximately true;
4. There is a sort of common-sense aspect to science, such that many of its core principles have been practiced by all people in all cultures;
5. Scepticism and self-criticism is an essential component of science

These assumptions seem pretty intuitive, and I think that to a large degree they follow from the ‘naive realism’ – to quote a term from one of my favourite philosophers and cognitive scientists – with which we view our world. But these assumptions are to varying degrees highly contentious, and they have been the subject of much debate amongst philosophers of language, science, mind, metaphysics, and so on.

1 and 2. The notion that we all live in the same ‘objective reality’ can be attacked in a number of ways.

The most powerful attack, in my opinion, comes from Wittgenstein. That attack claims that many, if not all, of the objects we observe to populate reality are only ‘objects’ because we have made them into objects through social practices, which he calls ‘language games’.  Language games aren’t necessarily about words, but rather are about the development of conventions or rules through context-specific practices (‘forms of life’). We learn to divide our world according to have we interact with it, and how we interact with it is learned and patterned. Other variants of the linguistic attack include Foucault’s ‘power-knowledge’, and certain awful post-structuralists whose names I shall not mention. While the strongest empirical formulation of this challenge, the Sapir Whorf Hypothesis, seems largely false, the underlying philosophical implications of the linguistic attack upon realism  – the epistemological and ontological aspects of it – make it difficult to imagine that any statement we could make, any proposition or predicate, could ever describe something ‘mind-independent’.

Another powerful attack comes from the instrumentalist tradition. This attack rests upon the notion that our observations are phenomenal: they are constructions of sense-data created by our brains, and as such, they represent reality but never grant us reliable or unmediated access to it. This tradition is very venerable, and rests upon Cartesian and Humean foundations. According to instrumentalists, all we do is divide up our sensations into useful and consistent patterns in order to effectively manipulate the world and solve our problems. While advances in cognitive science have shown that our brains appear to naturally divide sense data in certain ways, this doesn’t actually tell us anything about what is True, only what we can be reasonable confident will persist as phenomena.

There are, of course, excellent and interesting defences to theses attacks, but we should not be blithely assume that they succeed.

3.  The notion that we can develop approximately true knowledge of reality by minimising our biases or by following certain methods has many powerful challenges. Ever since the Duhem-Quine thesis – that we never falsify single theories but rather sets of theories – received broad (though not universal) acceptance in the 1950s, and Kuhn’s historical discussion of the many ways in which scientists are most definitely not sceptical, most definitely do not test all their assumptions, and most definitely do not progress in knowledge in any kind of linear way, scientific realism has been under attack.

Recently, realist philosophers of science have roughly divided into two general camps. The first, in the Lakatosian tradition, believes that while we don’t have the ability to ‘falsify’ individual theories in any kind of non-contextual way, we do have the ability to falisfy methodologies – substantive sets of assumptions about what the world is and how it can be studied – by seeing whether they allow us to accurately predict new things. The second believes that we can make true statements about cause and effect, and that we can possibly also divide reality accurately into entities based on their emergent properties, using a kind of principle of ‘inference to the best explanation’, or ‘abductive reasoning’.

However, anti-realist and instrumentalist philosophers have hit back hard. Larry Laudan, in ‘A Confutation of Convergent Realism’, has pointed out that the history of science is filled with theories that enabled highly accurate and new predictions, but whose central terms are now believed to be total fictions. Examples include phlogistons or aethers. Other philosophers have criticised the reliance upon emergence as being logically contradictory, and thus attacked the suggested that we could ‘abduce’ real entities out of our observations of cause and effect.

As in the previous case, all of these points are the subject of fierce and, in many cases, unresolved debate amongst our bright lights of ‘analytic’ philosophy.

4 and 5. We can’t even agree on what the principles of science should be in our culture. And to my knowledge, no other culture has produced such a vibrant debate over what they should be. However, it is true that basic principles of logic – non-contradiction, exclusion, and identification – appear to guide philosophical and ‘scientific’ debates in many other cultures than ours. As for scepticism…well, Wittgenstein has quite a bit to say about that too (language game!), as does Laudan (that it’s a context specific attempt to solve problems), not to mention many others. They’re all worth reading.

I’m going to suggest an alternative understanding of science, since it should be quite obvious by now that Natalie’s understanding of it is entirely culturally and philosophically contingent upon specific traditions. My alternative should allow us to still view science as a special process of enquiry  worthy of our esteem, and also give us grounds to criticise mysticism or fortune-telling as bankrupt. Here goes:

Science is a process of enquiring about the world that occurs within a community of people who a) engage in constant self and mutual criticism of the methods, assumptions, and conclusions of their enquiries and who b) produce ‘knowledge’ that is publicly accessible and c) in principle designed to clarify and disclose the world irrespective of any particular moral stance.

By this definition, we can criticise mystical claims about the world for being unwarranted on the grounds of their own methodological insufficiency (ie they are not reasonable even from within the system that produced them), and privilege science on the grounds that it is by definition something that is – or should be! – a public and politically neutral domain of enquiry, there to resolve disputes about how the world is regardless of how we think it should be. And crucially, we can criticise people for dismissing scientific knowledge when we have good reason to believe that it was produced according to methods that have a lot of potential for making our lives better – as Natalie has argued so pointedly.

Why study the philosophy of science?

‘The philosophy of science is as useful to scientists as ornithology is to birds.’ – Richard Feynman

So why should you study the philosophy of science, then? In particular, if you’re a scientist  then what is to gain from all this metatheory? How does it impact upon your day to day practice of research and theorising and so-forth?

Well, I can’t speak for you but I can speak a little bit about why I have found it helpful. Mind-altering, even.

So normally, most people think of science as trying to make valid causal inferences: the search for cause and effect. This is thought to occur via something called The Scientific Method whereby the scientist proposes a hypothesis and runs an experiment to see if the hypothesis ‘comes true’. If it doesn’t come true, the hypothesis is falsified and discarded. If the hypothesis seems really robust, we can start to call it a theory or even a law of nature. Even if one’s attitude isn’t quite so narrow as to what constitutes ‘science’, most still view prediction by induction to be the raison d’etre of scientific enquiry, where the subject of that enquiry are a reality that we can approximate in our theories and which exists in more or less the same form regardless of our ideas about it (eg when the tree falls in the forest and nobody is around, it sill makes a sound).

I am a social scientist. This means that I study what people do as part of society, and what sort of things characterise and constitute that society. It follows from the above definition of science that I should be looking for the causes and effects of social activity, and for general patterns of the type ‘whenever A happens, B follows‘. But people are really, really complex, and the methodologies available for studying them – those rules of analysis and problems or questions of interest that together define a research agenda – vary widely between fields and within them. Should I be trying to falsify hypotheses? Should I be using economic models? Should I be, I don’t know, just asking people to tell me why they do the things they do?

Let me give you an example of one of the puzzles that social scientists face; something that gives us cause to wonder if many of our common explanations involve some deeply paradoxical or counterintuitive assumptions. Consider the notion of social structures: if social structures affect our actions, but social structures only exist because of the actions of individual people, then doesn’t that mean that we’re both cause and effect? And if structures are more than the sum of their parts – us – how is that metaphysically possible? Isn’t that like saying 1+1=3? And if structures are nothing more than us, and can’t influence us, then what do they do? I’m not actually going to try to answer these puzzles – though I do have some ideas about them – but I do think they give you just a small glimpse into some of the conceptual difficulties that social scientists must face.

There is a critical immediacy to discussions on how to conceptualise notions like truth, causality, observation, explanation, and scientific progress when it comes to social science that don’t seem as pressing in the so-called ‘natural’ or ‘experimental’ sciences. That doesn’t mean they don’t matter deep down, but there doesn’t seem to be as much choice between radically different methods and assumptions for studying the same general thing. Some philosophers of science have suggested that the mark of a ‘mature’ science is this sort of methodological uniformity, but if that were true, then it would be grim news for social scientists, because I have the distinct sense that we’ll never get there. Or at least, we’ll get there only by radically changing the way we talk about people, in potentially impractical ways.

I think that a basic understanding of the philosophy of science – which I’ve already said is particularly important in social science – should consist of an understanding of the various debates on the following (interconnected!) issues:

• What are facts and what is a true statement?
• How do we find facts or determine whether a statement is true?
• What is the link between observation, theory, evidence, and truth? What gives us warrant to assert a claim?
• What is an explanation?
• How do we develop more ‘knowledge’ or make progress on our understanding and explaining?
• What does it mean to ‘do science’?

These are hard questions. Very hard questions. Of course, we will need to take up certain substantive positions on them if we are going to support our particular choice of methodologies, and of course I have my own opinions on them, but the only way to take a reasonable position is to know the basics of the various options, and to have some idea of where those options take you.

I won’t go into many specifics of who to read and why – at least not extensively – but  some excellent starting choices are Alexander Rosenberg‘s concise and cogent introductions to science and social science, Larry Laudan’s pragmatic theory of scientific progress, Laudan’s ‘Confutation of Convergent Realism‘ (for the slightly more advanced), and Peter Winch‘s short but powerful  ‘The Idea of a Social Science‘ (note: link is to an ebook and its only 160 pages!). I would highly recommend to anyone else in International Relations and political science to at the very least read Patrick Thaddeus Jackson’s book on how philosophy of science pertains to the field, though Jackson’s book would be a good start even for sociologists, anthropologists, critical theorists, etc simply for its breadth. Or just go through the readings in an introductory syllabus and see where they take you.

If you’re looking for some key insights right away, let me share a few with you:

• There are more ways to do science than to make hypotheses and test them by seeing if their predictions come true. While experiments of this nature can give us powerful ways of controlling the world, they may not give us the right kind of answer to other sorts of questions. For example, instead of determining which independent variables are stronger ’causes’ than others for ‘dependent variable’ effects, an event or outcome may be best understood as occurring when a whole set of factors come together in just the right way as to produce an effect. In other words, rather than a set of possible causes to find the real ones, the scientist studies one situation that they know was the real cause, and looks to see how it created the effect – in short, a search for causal mechanisms.
• The above alternative to the hypothetico-deductive method – ie experiments and hypothesis testing – is particularly pertinent to looking at very complex things, such as society or certain kinds of biology. That is because there are many causal factors, and it is very hard to know just precisely how they came together. Some of them might only need to be present, but others might do different things depending on their quantities and ratios. In a sense, some causal structures simply make something possible while others are part of a sequence in the process that leads to it happening in the way that it did.
• In the social sciences, quite often the sort of ‘explanation’ we are looking for is an answer to ‘why did X find it reasonable or appropriate to do what they did?’ This kind of explanation requires us to try to understand X’s situation in the same way X did, and look for the rules of behaviour or thinking that influenced X’s judgement. Often we can only really get an answer by deep, narrative study of X and X’s relation to their environment.
• There are powerful challenges to the notion that we could ever describe a kind of objective reality, not just in terms of cognitive biases but in principle. These challenges come from the nature of language (ebook), of consciousness, and of causality and metaphysics itself. They are worth studying in detail, because deciding how to deal with them is necessary to produce a coherent methodology. One way out is to be entirely instrumentalist, but this escape comes at great cost.

Even though my main area of study is International Relations and political sociology, I have found that studying the philosophy of (social) science has helped me think about how to view things like ‘states’ or ‘politics’ or ‘rhetoric’ or ‘power’ in hugely relevant ways, and to figure out what I can contribute in studying them.