The Beatles' "Old Brown Shoe"

This morning I listened to the second disc of the Beatles' Anthology 3, and I noticed a small thing about "Old Brown Shoe."  (It's in the single version too, but it's clearer in the version on Anthology 3.)  The melody of the vocal part descends at the end of the line "You know you pick me up from worse and try to drag me down," so you get the impression of the "drag[ging] down."