You will indeed be confused if you start with S3. That’s not to say you wouldn’t eventually figure out the basics, but you’ll miss the absolutely fascinating story, which contains a key whodunnit, of how they got there. S3 starts off at a run, and I think you’ll want to be up to speed.
In particular, you do NOT want to miss the scene in S2 where Allison Janney’s character gives Keri Russell’s character a succinct lesson in how to dress like a diplomat. It’s priceless, and I watched it over and over.
Honestly, it’s a great show. I wouldn’t advise missing a minute of it.