I think Bancroft's tendency to an actress-y quality got in the way of a character that was meant to be, in contrast to her self-dramatizing son, straightforward and rather ordinary. So the abruptness and off-the-cuff element in the character wasn't there.
And Harvey maybe was a little too old for the role by then.
But I agree, other than those two flaws, it kinda works.
