This phrase is believed to originate with the game of marbles, where some of the opponent's marbles could be kept upon victory. One could also just play for fun, which does not involve the risk of losing anything. The rules are typically decided at the start of a game and that is when the paticipating players establish if they are 'playing for keeps' or not... at least, that's how it works today.
Apparently, the game of marbles has been a fun pastime for both children and adults for several centuries now, possibly even longer. The expression has been used in the figurative sense of 'it's time to get serious' since at least 1842, where in the Hattiesburg American newspaper, it reads as follows:
"In the second and third heats, George realized this was a mistake, because it made Joe angry and he started playing for keeps."
* Mary was having fun on the field with a game of soccer, but once the other team scored, she decided the fun was over and it was time to play for keeps.
* Having lost a marathon race by just the hair on his chin, Thomas said next time he's going to play with keeps on his mind.
Note: The origins for most popular idioms are nearly impossible to find, if not completely! What's provided are the theories that may be plausible to how a phrase originated, and should not be taken as full confirmation.
In addition, the quotes that contain a phrase may be from old newspapers, poems, or books that were written hundreds of years ago, but one should not assume that just because a book from the 1850s uses a phrase, that the specific book is responsible for its origins. Perhaps the author heard it from some place else! Just keep that in mind.
Additionally, the quotes containing the expression are the oldest that I could find, so there may be an even older recording that I missed. If you are aware of any and want to correct me, feel free to contact me by e-mail.