If you really want to do something then why waste a "few years" doing something else? Honestly it doesn't matter if you're studying physics, programming or trying to invent the next slap chop -- what counts is that you have a passion for what you do and invest the time in that passion. Although be careful that your passion for programming is indeed that and not something else.
The "few years" would be spent finishing what I have started, it is a shame that my primary passion may be elsewhere but changing course now would set me back to far.
Well, it depends in what ways you're competing against Comp Sci majors, but even in their own major there is a lot of competition from competent physicists -- especially as we try to understand more and more about what quantum computers can and can't do. (Scott Aaronson's page and blog are wonderful to see what theoretical comp sci looks like; http://www.scottaaronson.com/ ).
If you just want to code, pick up a language, do it in your free time. CS majors will know a couple of more involved things; for example, many of them may have written a functioning, if minimal, compiler. If you want at least some familiarity with this, and if you can stand retro video clips with bad audio quality, you should watch the Abelson-Sussman lectures online: http://ocw.mit.edu/courses/electrical-engineering-and-comput...
I mention them in part because I think you might also like the ideas that Sussman brought back to his native engineering discipline; the above lectures are for a course called "The Structure and Interpretation of Computer Programs," but he later put out a book called "The Structure and Interpretation of Classical Mechanics," available here: http://mitpress.mit.edu/SICM/book-Z-H-4.html#%_toc_start
The point is that it doesn't have to be two divergent skill-sets. When I was at Cornell, they advertised their applied physics programme with the wonderful statement: "once you know the fundamentals, you're ready for everything." The simple habits you pick up as a physicist, like reasoning about the size of observable quantities or the cultivated habit of "whenever I hear a mathematical term I am going to look it up and read the definition until I understand what the hell they're saying" -- those can really become powerful when you start to write programs. Depending on what you're doing, it's maybe not as helpful as being fluent in the database language SQL, but SQL is much easier to learn and understand sometime later.
