As a boy, James was frail and prone to bouts of allergic attacks. He was neglected by his mother, who had been institutionalized following the death of her first son, John Jr., in 1897. James was constantly under the pressure of his grandfather, Mr. Howlett, who believed that James required constant punishment by a strong hand he used this to approve his abusive behavior. James spent his early years on the Howlett Estate with two playmates, Rose O'Hara, a girl who was brought in from town to be a companion to young James, and a boy named "Dog", Thomas Logan's son and thus James's half-brother.Thomas Logan, in a drunken state and armed with a shotgun, invaded the Howlett Estate with his son and attempted to take his former lover Elizabeth with
