The SMART program promises to raise your IQ by 20-odd points

schoolforthegiftedA write-up in The (Irish) Sun yesterday (posted on Twitter by Simon Dymond)  has brought a site called Raise Your IQ to my attention. It’s a brain-training site that makes a startling claim; that their SMART (Strengthening Mental Abilities with Relational Training) program can raise IQ, by an average of 23 points.  Those are some pretty big words, right there.

The business is a spin-out company from the National University of Ireland, and was started by Bryan Roche and Sarah Cassidy, of the psychology department at NUI. Dr Roche is an expert in a fairly niche area of psychological inquiry known as Relational Frame Theory. RFT is a theoretical framework that seeks to extend the radical behaviourist ideas of BF Skinner to encompass and adequately explain  complex cognitive processes (language, abstract thought, etc.). Skinner himself was firmly of the opinion that cognitive processes are in fact aggregations of lower-level behavioural responses, and that things like our sense of free-will, and our awareness of our own phenomenal consciousness are entirely illusory (views he expressed in his classic book Beyond Freedom and Dignity, as well as in many more academic works). RFT is essentially a neo-Skinnerian approach which takes into account the accumulated evidence for complex (i.e. nominally non-associative) cognition.

As a theoretical approach RFT has some interest then, however Dr Roche has managed to add a practical twist and come up with something marketable, dubbed ‘relational training’. Essentially it’s a set of questions and tests that are based on relationships between different objects, words or categories. Some examples are below (pasted from The Sun article):

LEVEL 1: If a broad bean is a type of bean, is a bean a type of broad bean?

LEVEL 2: If an Alsatian is a type of dog, and a dog is a type of mammal, is an Alsatian a type of mammal? Is a mammal a type of Alsatian?

LEVEL 3: If yesterday I felt sad and today I feel happy, and if now was then and then was now, how did I feel then?

LEVEL 4: “Car” is the English word for the Swedish word “Bil”. “Car” is also the English word for the Italian word “auto”. “Coche” is the Spanish word for the Italian word “auto”. What is a Coche in Swedish?

Readers who were fortunate enough to have endured a Classical education will immediately recognise the broad form of these as a kind of logical puzzle known as a syllogism, the formal description of which dates back to at least Aristotle; plus ça change. Syllogistic reasoning is also a mainstay in various IQ-type tests.

So, what about these pretty radical claims of raising your IQ by 20 or even 30 points? Does it work? What’s the evidence? As usual I’m afraid, the answer is ‘scant, to non-existent’. A prominent page on the Raise Your IQ site is titled ‘Scientific Evidence’ and discusses in some detail this paper by Cassidy et al (2011; PDF), which does indeed show some impressive effects in two separate experiments. However, experiment one had only four participants in each group (training vs. control), and experiment two had only eight participants (with no control group). These are tiny samples and (as has been extensively discussed, very recently) small samples can lead to the ‘winner’s curse’; an inflation of the apparent effect size. The effects reported in this paper are very large indeed, but honestly, I’m surprised the reviewers didn’t flag up the fact that you just can’t do ANOVAs with four data points! It’s a massive violation of the assumptions of the central limit theorem. In addition, the control group in experiment one was an ‘inactive’ control i.e. one that received no training at all, rather than some kind of placebo training.

The rest of the papers listed on the Scientific Evidence page of the site appear to be fairly theoretical, or only tangentially related to the SMART program. The available evidence from the Cassidy et al. (2011) paper is wafer-thin, compared to the startlingly bold claims being made. One other thought that occurred to me is that because the form of the training is so similar to the form of some questions used on many IQ test, it might represent a form of training-to-the-test; practicing IQ tests makes you better at performing IQ tests – well, duh.

I find myself unable to completely condemn these guys though. Maybe I’m going soft, or maybe I’m just a sucker for some good old-school radical-behaviourism-based interventions. The program does have the (somewhat unusual) virtue of being based on a fairly coherent (if not widely-accepted) theoretical foundation. This doesn’t change the fact that the big claims it makes are wildly out of sync with the available evidence, and in this sense, it’s precisely the same as all the other brain-training neurobollocks-merchants out there fighting for a slice of the gullible consumers cash.

 

 

Advertisements

11 responses to “The SMART program promises to raise your IQ by 20-odd points

  1. Much as I suspected. It would have been nice to see how individuals IQ tested elsewhere and then trained, fared on the training by being tested again. And then see if that training ‘cemented’ itself in the long term.
    If it give you a test and you do badly on it, then give you another and you do well, has you IQ increased!
    Nice post.

  2. The other curious thing about the whole “raise your IQ thing” is the notion of IQ as a kind of cultural capital. As I’m sure you know IQ was an invention by Binet to try and root out children who needed special education in early 20th century France. Even if their claims were true what would be the point of boosting your IQ by 20 points? An IQ of 145+ will not get you a job with NASA, any more than being seven foot tall will get you onto the national basketball team. More to the point, unlike with cultural capital, where it’s socially acceptable to boast of it in conversation (“I don’t watch the x factor, I go to the opera”, “I was into them before they were famous” etc.), it’s seen as extremely rude and vulgar for a person to publicly mention their IQ score, you either come across as being intelligent or you don’t, if you get to the point in a conversation where you feel compelled to literally say “f*** you I have an IQ of 192”, you’ve probably long since lost the argument.

    • Excellent points Seanan – there’s lots of issues with the whole idea of IQ testing which I’m sure many readers will be aware of. And yes, raising your IQ in casual conversation is definitely a douchey move!

  3. The scepticism of the author is well grounded on statistical and (to a lesser extent) RCT-friendly assumptions – where the strength of the program lies in its emergence from basic behaviour analytic findings (read Relational Frame Theory, circa. 2001), which points to a solid background of data. The coherence of an empirical theory with an applied procedure is unusual but should nonetheless be regarded as of fundamental importance. I assume the author of the article is familiar with the benefits that small-n multiple-baseline changing criterion designs can offer for measurements along a single unit/dimension of behaviour (see Cowan, Hennessey, Vierstra, & Rumrill, 2004 for a recent appraisal), sometimes more so than large-n or quasi-experimental designs.
    If we consider IQ as a cultural construct for assessing skills believed to be of predictive value for performance along other socially derived measures of acquired verbal repertoires (read ‘intelligence’), two questions are commonly asked – 1. Does IQ really lead to a better job/life/partner i.e. what is it’s external validity? This is the matter of public and academic debate, and it is a valid one. A second question not so openly discussed could be 2. What skills do IQ tests measure? An analysis of the said skills demonstrates that a majority of these are directly trainable ‘core’ verbal processes that have been shown to generalize along multiple societal and neurophysiological contexts.
    One may recall that Watson & Raynor (1920) reported a ‘fear conditioning’ study with a n of 1 which served to launch a host of clinical interventions and therapies ranging from in vivo exposure therapy to a behavioural pharmacology. All that is intended to illustrate is that while the oft-assumed predictive validity of a scientific method is proportional to higher power/large effect sizes according to many, this by no mean denounces the conceptual value of solid experimental findings.

  4. As the original blog outlined, the RaiseYourIQ method (SMART training) has emerged form a neo-Skinnerian paradigm – known loosely as “modern behavior analysis” or more precisely as contextual behavioral science (http://contextualscience.org).

    Behavior analysis has been supremely successful in developing treatments for every manner of behavioral and intellectual difficulty, using low n studies, and it is from the procedures used in that tradition that the SMART program has emerged. Like many behavioral methods, it has not come from a large RCT study, but has evolved through a mix of clinical practice and a range of studies showing that relational skills can be trained (dozens of studies) and that they are both related to a wide range of intellectual functions, and that intellectual skills of various kinds can be improved with relational skill training (most of these studies are with reading and vocabulary skills but at least one has used it to improve mathematics).

    Actually Relational Training teaches what was referred to as syllogistic reasoning but to a set of criteria that actually make it quite more complex – a level NOT reached by most individuals in our culture, unless they are logicians. The correct term is “stimulus equivalence” which is a mathematical concept that extends the range of conditions that need to be met over and above mere derivation between first and third terms. Specifically, in a syllogism; If A=B and B=C, it can derived that A=C. But in stimulus equivalence it is also necessary to show that the individual can derived that B=A, C=B (symmetry), A=C (transitivity). We usually go a step further and teach kids to derived that C=A (combined symmetry and transitivity). It turns out that kids DO need to be taught these skills (numerous studies on this), and that it CAN be taught – and that it underlies all language ability. In fact, emergent stimulus equivalence is so synonymous with language acquisition, that we refer to it as a “verbal process”. Deriving relations is language itself, and has never been satisfactorily shown in animals (plenty of studies on this too). So Relational Frame Theory (RFT) HAS identified some basic units that we think of as the building blocks of intellectual activity. These could never have been found in brain research alone because they exist only as functional behavioral units – although behavior analysts have now studied the neural correlates (e.g., http://onlinelibrary.wiley.com/doi/10.1901/jeab.2007.93-05/abstract)

    BUT – it gets more complex – Relational Frame Theory studies new behavioral units never studied by the ancient Greeks. For example, the relation of “oppositeness” is crucial in mathematics, for understanding concepts like negative numbers. The understanding does not come first – the behavioral unit does. Once the individual LEARNS to respond to C as the same A given A Opposite to B and B opposite to C as conditions, they are able to derive SAME relations from other topographically varying forms of combined OPPOSITE relations. We have published dozens of studies showing how the word OPPOSITE itself acquires its contextually controlling functions over the other words in the argument in the first place, so this is not a tautology. And it goes on – the opposite of an opposite of an opposite is in fact an opposite, not a same relation. And there are more relations that we have identified as important for other intellectual skills – such as temporal ones: If A is before B then B is in fact after A (the relation is not symmetrical). Hierarchical ones: If A is a type of B the B is NOT a type of A. Deictic ones: If I were you and you were me, then you would be here and I would be there (crucial for spatial reasoning). And so on. None of these latter relational skills are syllogistic. It is THIS broad skill set that SMART trains – and all it is doing is drawing together the various practices and evidence form the field and offering them all in one package. Of course we use very precise teaching technologies based on behavioral principles. The effectiveness is all in the delivery system.

    So it is correct that syllogistic reasoning IS a crucial part of the skill required to do well in an IQ test (e.g., analogy tests) – but this is why we teach it using SMART (see this paper – http://psychology.nuim.ie/sites/psychology.nuim.ie/files/images/Cassidy%20et%20al%2011.05.pdf). There is no “trick”. SMART simply teaches the core skills a person needs to be good at to do well in school and work and daily life. The amazing bit, is that we HAVE hit on the basic building blocks of reading, and mathematical understanding, and that is the breakthrough (if you can call 20 years of work across a dozen labs a breakthrough).

    Behavior analysts use small n studies due to clinical tradition, but also due to the use of non group designs, like elegant multiple baseline designs. The behaviors we are trying to produce are so complex that n’s of 1-3 subjects are still sometimes published. For example, consider this level of behavioral control typical of a study in the area of Relational Frame Theory. A participant is required to press 5 of 10 coloured keys on a keyboard in a specific sequence given a rule made up entirely of nonsense words. The nonsense words have no function (i.e., meaning) at the start of the experiment. By the end of function training the participant can press the correct five keys, in the correct order, on 32 trials in a row (i.e., 32X5 correct consecutive key presses). Now this is reported in a paper as a demonstration of the effectiveness of the training for establishing the behavior pattern of interest. But consider the appropriateness of asking that researcher to prove by inference that the 5X32 key press sequence was not by chance. It flies in the face of clinical and laboratory judgment and puts inference before expertise. We can see on the learning curves the clear and slow emergence of the sequence as a trajectory – we can even manipulate it by altering the reinforcement contingencies so that the functional relationship between the training and the behavior of interest is highly apparent. Inference adds noting in this case, and a second n does not make the first instance of behavioral control any more or less impressive if behavioral control was demonstrated WITHIN the first participant across various stages of the training. Reasons for obtaining less or more behavioral control with another individual may be due to variables that do NOT APPLY to all individuals – and inference will only mask this. The aim is to get control over EACH individual’s behavior, NOT NECESARRILY using the same method in each case, only the same principles. This in essence, is applied behavior analysis (ABA). So additional participants are never viewed as providing further evidence for an effect because the group is getting larger, but because the number of replications using an n=1 is getting larger – this is crucially different to standard group design logic, which we think of as lazy and often leading to terrible behavioral control but impressive statistical behavior on the part of the observer. I guess this is why brain training has never worked – the focus was never on identifying the necessary behavioral units and so good control was never possible (e.g., the process according to which the n-back procedure helps to increase fluid intelligence, if that is reliable, is almost of no interest to the original researchers, and it appears to us to be of far too little concern to everybody else – the effect is statistical – the behavioral process is a mystery!).

    The level of control required by behavioral studies needs to be borne in mind when one considers them. The Cassidy et al paper DID also require extended and complex response sequences, in which subjects were required to pass dozens of test by pressing one of two correct keys given 16 different relational probe questions in a row, without error – dozens of times over in some cases, before they met criteria to be considered to have shown a relational skill improvement. THIS was where the control came in – the interesting emergence of a rise in IQ was a secondary phenomenon mediated by the impressive behavioral control shown.

    In the Cassidy et al. study it was not crucial to have a control group because each participant’s baseline IQ served as a baseline against which to compare increases, which were in all cases above the natural increases expected or explicable by psychometricians (e.g., 10 points or so). It is surprising that some researchers require a piece of software to INFER that 8 IQ rises (experiment 2) all in the same direction, in a study that took months of intervention, were of psychological importance. Put simply, your IQ score does not leap 15 points by “chance” – the test-retest criteria established by the test manufacturers control for that. So there is effectively no chance that ALL of the IQs of the Cassidy et al. participants did rise by chance. Unless we know that this does sometimes happen – we do not need to have a control group to make an in-principle demonstration. Nevertheless, I am sure in time, and I hope that such larger studies will be done.

    So it is a little naïve to assume that all effective treatments come from large n studies based on a Popperian hypothesis testing paradigm. Generally behavior anlaysts avoid theory – because we are inductive in our approach to science – and as a result – we eschew hypothesis testing (a method that can only arise from hypothethico-deductive reasoning). Instead, we insist on behavioral CONTROL – and we do not ignore even one individual absence of effect in even one participant. Large n studies are designed to ignore high failure rates across individuals. So a headache pill may be found to be effective in a large RCT, when in fact it works only on 60% of the people. Behavior analysts would prefer high levels of control over the behavior of every single individual – with every variation accounted for (variation IS our subject matter). So we make up in precision what we lack in P values.

    Incidently, we also use THREE, and not just one criterion for an effective study outcome – we seek not just precision in our data, but also DEPTH and SCOPE. This further reduces the emphasis on p and n.

    The claims we are making at RaiseYourIQ are the result of a myriad studies all triangulating in on the same conclusion – this is a far healthier way to draw a conclusion than have a single large n study. Anyone who thinks otherwise must believe in the elusive Exprimentum Crucis and adopt some kind of naïve realist or even logical positivist approach. Our philosophy of science is a form of pragmatism known as Functional Contextualism.

    We believe that science is an open and self correcting process and we encourage people to use our tool to assess it for themselves – we will always believe the evidence.

    Finally, but crucially, behavior analysts are highly skeptical of the concept of IQ and we wish to dismantle and replace it with a functional unit called the Relational Ability Index (RAI). IQ is a hypothetical construct with a dubious history (ala Francis Galton, the Stanford-Binet test and so on), and is used wrongly in many cases. We have written extensively on this in our own journals and in books. So, in fact, while we appeal to IQ rises in our public information, what we are really concerned with are improvements in intellectual skills as broadly defined. We will always endeavor to improve each and every skill regardless of what the culture defines as IQ at any given point in time.

    So we agree with one commentator above, that it is of little use to merely raise your IQ as a test score – but SMART is not doing that – it is enhancing repertoires of intellectual skills that over time increase the fluency (accuracy and speed) of complex responding in a range of situations, so that IQ test items also reflect that shift. The IQ rises we see (in every case we have tested so far in published and unpublished research) might be considered an epi-phenomenon. We do not target it directly. Drawing attention to it in public information is simply a way to speak in a language an typical individual understands. But as we change the way we think about intelligence, exorcise it as a hypothethical construct and replace with a functional measure that emphasises variance over contrived stability, we may one day be able to call our website “RaiseYourRelationalSkills.com” and every man and woman on the street will know what we mean.

    For interested readers, here is a recent book on the topic.
    http://www.amazon.com/Relational-Responding-Applications-Developmental-Disabilities/dp/1572245360

    • Dear Bryan,

      Many thanks for taking the time to make such detailed comments – lots to go through and think about here! Thanks for clearing up my misunderstandings about the program and expanding on its theoretical basis and practical instantiation – very useful. I take the point that it’s obviously a lot more than basic syllogism.

      I also take the point about small N studies. I have something of a background in psychophysics, so am well used to reading papers with only a few subjects in (a couple of which are sometimes the authors!) – it’s a similar kind of experimental approach to what you outline above. I believe my original point still has value though; you are making the case for a clinical-like intervention, and to make that case persuasively I think you’ll need to follow standard RCT-type protocols (placebo-training control conditions, double-blinding, etc.). I understand that each participant in the published study represented a large amount of data, and therefore the intra-subject noise/variance was probably exceedingly low, but that doesn’t change the fact that to demonstrate group effects convincingly you need more than four subjects! If only for the reason that such a low N violates the assumptions of pretty much every standard statistical test. I realise you’re making a case for the usefulness of a less standard Null-Hypothesis-Significance-Testing approach here, and there’s definitely something in that argument, but to be really convincing I think you’ll have to engage with the standard methods.

      If the (extremely large!) effect seen in the small study is really genuine, it should be no problem demonstrating it in a larger population, with RCT-like measures; you can only gain, by adopting such methods.

      >Put simply, your IQ score does not leap 15 points by “chance” – the test-
      >retest criteria established by the test manufacturers control for that. So
      >there is effectively no chance that ALL of the IQs of the Cassidy et al.
      >participants did rise by chance.

      True. But without RCT-like studies, you can’t rule out confounding factors. For instance, it could be that the kids IQ improved simply because they received long periods of individual attention, or just because they practiced general computer skills. This is the benefit of ‘active’ placebo conditions – similar interventions, but with the hypothetically-crucial component removed.

      Many thanks once again for the excellent dialogue, and I look forward to seeing more studies on your program when they’re available – if you do publish anything else in the future, please let me know and send me a copy.

      Cheers,

      NB.

  5. Pingback: My Homepage

  6. Pingback: How to develop and market your neuro-product | NeuroBollocks

  7. James Flynn’s book “What is Intelligence?:Beyond the Flynn Effect” discusses the effect of interventions for increasing IQ in depth. Flynn found that to sustain increases in IQ scores the stimulus of intervention had to continue. Without the stimulus of the intervention IQ scores eventually reverted. Flynn studied 3 generations of an Asian cohort and the shifts in average IQ related to environment of his groups of subjects. He also discusses his research at length as well as other IQ research and the nature of IQ tests and the relation to the Flynn Effect.

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s