I picked up the mystery seed/fruit/nut from the yard and got some more pictures. It was suggested that it was probably a walnut but I don't know after this.
Some measurements. These were a little bit of a challenge. You'll see why.
This is after I cut into it a little. It smells like a peach inside, which is odd.
It's hard to cut and has a very hard center, like a pit. It smells like a peach when you cut it open. I still don't really know what it is but Trudy is convinced it was the latest cat toy in the house. She meowed angrily when I wouldn't let her bat it around very much.
EDIT: We figured it out! It's a pignut hickory. I didn't have any hickory trees around our house when I was growing up so I wasn't familiar with them.