As an applied ethologist, I’m additionally very interested in the KINDS of questions in the survey. I participated at the time and reviewed the questions two days ago. Frankly it’s logical to me that even without the NOISE you describe, that we would not expect to observe much difference between breeds in the behaviors asked about in the study, because the vast majority (literally almost all) off the questions aren’t targeting behaviors that had selective pressure on them. Dogs are different because of the selective pressure humans have put on their behaviors historically for functions- so questions that get at THOSE distinctions are what I would expect to demonstrate the variations in answers. The questions they did ask about breed/clade specific motor patterns (especially relative to modifications to the predatory sequence) DID demonstrate variation. But that was brushed over in the conclusions. Since there was no selective pressure on behaviors like paw crossing, circling before pooping, licking a bowl after a meal, or being scared of strangers (no one ever bred dogs to display these behaviors deliberately) - why would we expect to observe variation? Had the questions targeted the meaningful differences we know about between genetic groups of dogs, I think we would have seen much greater trends in the data even with the noise you mention. Can’t wait for your full piece!
Informative as always Stuart! As an aside your (very sweet looking) dog is almost identical to my pal Tommy’s dog, which I believe is a cockapoo as well.
I am enjoying your critiques, careful and thorough as they are. I also have noticed the trend among news reports (and much of Twitter!) to hype or misread papers. It is imperative to read the original article.
hiya subscribing as also write perhaps good about definitely terrible science. here is my latest on the rabbit hole we've fallen right down into with Alice. We seem to be sitting at a mad virologists tea party https://georgiedonny.substack.com/p/spikes-and-knobs?s=w
Lab assessment might indeed be better for behavioral measurements, but the poor performance results of in-shelter assessments make me wonder if that would really be the case.
At any rate, thank you for devoting attention to this!
---Stuart misrepresents major sections, for example the ANOVA was also performed on the ~6,500 sample 'candidate breed' sample and found the same results (actually about 30% lower). He also ignores the analysis of admixture in mutts using the lmer approach. Both give extremely concordant results.
---Pretty much every large study of dogs uses survey responses from owners, even the dissenting and much worse paper that Ritchie cites.
I am sure you could address these critiques of your critiques.
Sure! The ANOVA was indeed also performed on the n=5,500 (whoever wrote 6,500 has gotten the number wrong - just take a look at the Data S9 spreadsheet) candidate breed sample. But as the overall n has increased, so has the number of breeds, so in this case we have 121 breeds instead of 77, which means you get about 40 dogs per breed. Don't get me wrong - that's better than a sample of 27. But it's not THAT much better than 27. And, of course, we're less sure about whether they're that breed. And of course the same issues apply with the measurement error.
Also, 30% lower? I saw this in the paper, but (like I note in the post re the other percentages) was confused about it, because for the behavioural factors at least, I get 8.7%, which isn't 30% lower than 9%. Again, it's all a bit difficult to work out where the numbers are coming from. In any case, the 9% number was the one they reported to the press, so it's the one I focused on here.
For the lmer/admixture part - yes, the behavioural part of the admixture result does indeed give a similar number. BUT, the % variance explained by breed for *physical* traits is dramatically lower: from the admixture analysis it's like 20%, but in the phenotypic analysis it's like 50%. So there's something weird and inconsistent here, and it isn't "extremely concordant" if you step back and look at the results overall. I think it's safer to say, well, those admixture results are hampered a fair bit by the sample size so won't be super-reliable, and leave them out of the discussion - which I did for this post, and which the newspaper stories did too.
And yes, I agree that every large study of dogs uses survey responses. This is why I said the "controlled conditions" thing was far less likely to happen than the more pragmatic method of just asking dog owners. But if you're going to use a noisier measure, it would be better to have far bigger sample size per breed - that's my point.
I’m a smidge less skeptical than you about how much noise you’d expect in the owner surveys. At least my priors are that people have a pretty clear sense of how their dogs differ from other dogs, so I’m prepared to believe a carefully designed survey needn’t have so much noise that you’d miss a main effect of breed on personality at those cell sizes.
But I can’t find a power estimate in the paper, so who knows. What surprises me a bit more is I also can’t find any controls for early socialization (e.g., age of pet adoption). Intuitively I wouldn’t expect to detect much effect of breed on personality before accounting for that variation. (I’m thinking back to those 90s Michael Meaney epigenetic studies with the rodent adoptions influencing personality.) Maybe there wasn’t much variation in socialization among the purebreds.
Unfortunately it isn't a particularly well designed survey (many of the statements are ambiguous and poorly worded - e.g. "my dog is a people person"), and the average dog owner is notoriously bad at evaluating their dog - as most dog trainers and behaviourists would attest. Furthermore, the statements are often confounded by the fact that dog owners rarely have the ability to put their own dog's behaviour into a broader perspective: I have a hunting spaniel so most other breeds seem borderline lethargic by default, and conversely what is normal for me is hyperactive for someone else. Multiply that by every other breed there is...
And you rightly point out that age of adoption isn't included, nor is origin of dog: many of the statements refer to behaviours or traits that can be and are often caused by prenatal/neonatal experiences, early life socialisation, living environment, owner's ability to train the dog and their relationship with the dog (which every dog owner thinks is GREAT even if every sign suggests it isn't according to the dog).
And finally: there is other literature that does indeed suggest that, as Stuart links in the post.
May 1, 2022·edited May 1, 2022Liked by Stuart Ritchie
Thanks, those are all really helpful points. I haven’t seen the actual instrument but I’m equally prepared to believe it’s not terribly rigorous.
Part of my initial reaction was about an assumption I run across a lot, which is that survey scales are necessarily noisier than physiological measures. But of course there’s no denying survey scales can be really messy and researchers can read way too much into them.
It’s such a fascinating question about how accurately people read pets’ personalities. That’s an interesting idea about how your own pet influences your baseline. Low priority for research I guess, but it’d be so fun to see studies of it and compare that to the literature on how accurately people perceive human personalities.
Humans are slightly different because it's our own species so we can introspect and we have much more experience over a lifetime of working and interacting with others of our kind - most people only have experience of a handful of dogs at the END of their lives, because most people have just one at a time, and unfortunately the majority of dog owners make minimal effort to learn about dogs (communication, needs etc). What one person thinks is shy might actually be an anxious, shut down dog...
I've even had experienced veterinary oncologists characterise my dog "hyperactive" when, from my perspective, she was not particularly lively, and certainly very average activity for her breed... It's relative :) Like I said, most other breeds seem lethargic to me - if my dogs walked as slowly as the majority of dogs I see in the forest, I'd rush them to the vet immediately because they only move that slowly if they're very ill.
The statements also include...
"Dog enjoys life"
Who dares respond their dog doesn't?!
"Dog moves normally"
What's normal? The normal for one dog (breed and individual) is abnormal for another as even their structures vary enormously.
"Dog seems to get excited for no reason"
Says who? The human who has vastly inferior sense of smell?
The human who controls the dog's entire life - even seemingly small things can be exciting if your life is really dull... how judgmental to say "for no reason", like the benchmark for dogs is to be seen but not heard, to wait on the sides like a decorative robot brought out when it suits humans.
Interesting! The challenges to personality judgment of humans, of course, are that we're good at impression management and our behavior varies so much based on our social roles. So intuitively I'd have guessed people are better at judging pet personalities. But you've definitely convinced me of our limitations. And now that you point out some of the individual items ("dog enjoys life"?!), I'm wondering how much personality variation they missed due to ceiling effects.
As an applied ethologist, I’m additionally very interested in the KINDS of questions in the survey. I participated at the time and reviewed the questions two days ago. Frankly it’s logical to me that even without the NOISE you describe, that we would not expect to observe much difference between breeds in the behaviors asked about in the study, because the vast majority (literally almost all) off the questions aren’t targeting behaviors that had selective pressure on them. Dogs are different because of the selective pressure humans have put on their behaviors historically for functions- so questions that get at THOSE distinctions are what I would expect to demonstrate the variations in answers. The questions they did ask about breed/clade specific motor patterns (especially relative to modifications to the predatory sequence) DID demonstrate variation. But that was brushed over in the conclusions. Since there was no selective pressure on behaviors like paw crossing, circling before pooping, licking a bowl after a meal, or being scared of strangers (no one ever bred dogs to display these behaviors deliberately) - why would we expect to observe variation? Had the questions targeted the meaningful differences we know about between genetic groups of dogs, I think we would have seen much greater trends in the data even with the noise you mention. Can’t wait for your full piece!
Informative as always Stuart! As an aside your (very sweet looking) dog is almost identical to my pal Tommy’s dog, which I believe is a cockapoo as well.
Funny you should say that…
I am enjoying your critiques, careful and thorough as they are. I also have noticed the trend among news reports (and much of Twitter!) to hype or misread papers. It is imperative to read the original article.
hiya subscribing as also write perhaps good about definitely terrible science. here is my latest on the rabbit hole we've fallen right down into with Alice. We seem to be sitting at a mad virologists tea party https://georgiedonny.substack.com/p/spikes-and-knobs?s=w
Lab assessment might indeed be better for behavioral measurements, but the poor performance results of in-shelter assessments make me wonder if that would really be the case.
At any rate, thank you for devoting attention to this!
I linked this post on Reddit https://www.reddit.com/r/genetics/comments/ug2iur/humans_cant_quit_a_basic_myth_about_dog_breeds/
and got this:
---Stuart misrepresents major sections, for example the ANOVA was also performed on the ~6,500 sample 'candidate breed' sample and found the same results (actually about 30% lower). He also ignores the analysis of admixture in mutts using the lmer approach. Both give extremely concordant results.
---Pretty much every large study of dogs uses survey responses from owners, even the dissenting and much worse paper that Ritchie cites.
I am sure you could address these critiques of your critiques.
Sure! The ANOVA was indeed also performed on the n=5,500 (whoever wrote 6,500 has gotten the number wrong - just take a look at the Data S9 spreadsheet) candidate breed sample. But as the overall n has increased, so has the number of breeds, so in this case we have 121 breeds instead of 77, which means you get about 40 dogs per breed. Don't get me wrong - that's better than a sample of 27. But it's not THAT much better than 27. And, of course, we're less sure about whether they're that breed. And of course the same issues apply with the measurement error.
Also, 30% lower? I saw this in the paper, but (like I note in the post re the other percentages) was confused about it, because for the behavioural factors at least, I get 8.7%, which isn't 30% lower than 9%. Again, it's all a bit difficult to work out where the numbers are coming from. In any case, the 9% number was the one they reported to the press, so it's the one I focused on here.
For the lmer/admixture part - yes, the behavioural part of the admixture result does indeed give a similar number. BUT, the % variance explained by breed for *physical* traits is dramatically lower: from the admixture analysis it's like 20%, but in the phenotypic analysis it's like 50%. So there's something weird and inconsistent here, and it isn't "extremely concordant" if you step back and look at the results overall. I think it's safer to say, well, those admixture results are hampered a fair bit by the sample size so won't be super-reliable, and leave them out of the discussion - which I did for this post, and which the newspaper stories did too.
And yes, I agree that every large study of dogs uses survey responses. This is why I said the "controlled conditions" thing was far less likely to happen than the more pragmatic method of just asking dog owners. But if you're going to use a noisier measure, it would be better to have far bigger sample size per breed - that's my point.
seems he responded again https://www.reddit.com/r/genetics/comments/ug2iur/humans_cant_quit_a_basic_myth_about_dog_breeds/i7alwfg/?utm_source=share&utm_medium=ios_app&utm_name=iossmf&context=3
it’s typical bitching from Kevin Bird lol
can you respond to kevin’s kvetching https://twitter.com/thebirdmaniac/status/1520804254637821954?s=21
I’m a smidge less skeptical than you about how much noise you’d expect in the owner surveys. At least my priors are that people have a pretty clear sense of how their dogs differ from other dogs, so I’m prepared to believe a carefully designed survey needn’t have so much noise that you’d miss a main effect of breed on personality at those cell sizes.
But I can’t find a power estimate in the paper, so who knows. What surprises me a bit more is I also can’t find any controls for early socialization (e.g., age of pet adoption). Intuitively I wouldn’t expect to detect much effect of breed on personality before accounting for that variation. (I’m thinking back to those 90s Michael Meaney epigenetic studies with the rodent adoptions influencing personality.) Maybe there wasn’t much variation in socialization among the purebreds.
Unfortunately it isn't a particularly well designed survey (many of the statements are ambiguous and poorly worded - e.g. "my dog is a people person"), and the average dog owner is notoriously bad at evaluating their dog - as most dog trainers and behaviourists would attest. Furthermore, the statements are often confounded by the fact that dog owners rarely have the ability to put their own dog's behaviour into a broader perspective: I have a hunting spaniel so most other breeds seem borderline lethargic by default, and conversely what is normal for me is hyperactive for someone else. Multiply that by every other breed there is...
And you rightly point out that age of adoption isn't included, nor is origin of dog: many of the statements refer to behaviours or traits that can be and are often caused by prenatal/neonatal experiences, early life socialisation, living environment, owner's ability to train the dog and their relationship with the dog (which every dog owner thinks is GREAT even if every sign suggests it isn't according to the dog).
And finally: there is other literature that does indeed suggest that, as Stuart links in the post.
Thanks, those are all really helpful points. I haven’t seen the actual instrument but I’m equally prepared to believe it’s not terribly rigorous.
Part of my initial reaction was about an assumption I run across a lot, which is that survey scales are necessarily noisier than physiological measures. But of course there’s no denying survey scales can be really messy and researchers can read way too much into them.
It’s such a fascinating question about how accurately people read pets’ personalities. That’s an interesting idea about how your own pet influences your baseline. Low priority for research I guess, but it’d be so fun to see studies of it and compare that to the literature on how accurately people perceive human personalities.
Humans are slightly different because it's our own species so we can introspect and we have much more experience over a lifetime of working and interacting with others of our kind - most people only have experience of a handful of dogs at the END of their lives, because most people have just one at a time, and unfortunately the majority of dog owners make minimal effort to learn about dogs (communication, needs etc). What one person thinks is shy might actually be an anxious, shut down dog...
I've even had experienced veterinary oncologists characterise my dog "hyperactive" when, from my perspective, she was not particularly lively, and certainly very average activity for her breed... It's relative :) Like I said, most other breeds seem lethargic to me - if my dogs walked as slowly as the majority of dogs I see in the forest, I'd rush them to the vet immediately because they only move that slowly if they're very ill.
The statements also include...
"Dog enjoys life"
Who dares respond their dog doesn't?!
"Dog moves normally"
What's normal? The normal for one dog (breed and individual) is abnormal for another as even their structures vary enormously.
"Dog seems to get excited for no reason"
Says who? The human who has vastly inferior sense of smell?
The human who controls the dog's entire life - even seemingly small things can be exciting if your life is really dull... how judgmental to say "for no reason", like the benchmark for dogs is to be seen but not heard, to wait on the sides like a decorative robot brought out when it suits humans.
Yeah. Not a fan of the statements :)
Interesting! The challenges to personality judgment of humans, of course, are that we're good at impression management and our behavior varies so much based on our social roles. So intuitively I'd have guessed people are better at judging pet personalities. But you've definitely convinced me of our limitations. And now that you point out some of the individual items ("dog enjoys life"?!), I'm wondering how much personality variation they missed due to ceiling effects.
Great comments - thanks both!
"Dog enjoys life" is great and very cute, but my favourite is Question 76: "Knows is a dog".