In short: Caltrain checks tickets enough that you cheating the system isn't worth it.
This post is from about a month of data in 2012, so if things've changed, keep that in mind.
Growing up in Palo Alto had me riding Caltrain a fair bit. Every so often, they checked tickets, but they didn't check every time. I was curious: how often do they check? Is it ever worth not buying a ticket and risking it, despite the fact that you'd be stealing?
The answer is no: best-case scenario, it's just under 10 times more expensive.
I rode the Caltrain to work most weekdays during my internship this summer, and I wrote down whether they checked tickets. This was to and from Zone 1 and Zone 3. I tracked whether I left in the morning (arriving in San Francisco between 8:15pm and 8:50pm) or the evening (leaving San Francisco between 4:30pm and 7pm).
First of all, the highest Caltrain fare is $12.75, and the first citation is $250 at most, which looks to be a fairly typical amount, according to some Googling.
Caltrain checked tickets 14 out of 31 times I recorded, which is about 45%. If I paid the maximum possible fare 31 times, I'd pay $395.25. If I had no ticket and paid all the fines, I'd pay 45% of $7750 (the amount I'd pay if I got the fine every day), which comes out to $3487.50. It's almost 10 times worse!
How about times of day? My data found that they checked tickets in the morning about 57% of the time, which is even worse. They only checked about 35% of the time in the evening, so it'd be slightly better, but still not worth it -- it's about 6 times worse.
My data wasn't too scientific and I could've gathered more data (specific time of day, the train number, between what stops they checked, more weekends, specific fine research), but the results are so far in the "don't cheat" direction that I suspect it's never worth it.
So...don't cheat the Caltrain!
If you'd like to take a look at my data, here it is!