Everytime the SF pulls up for a shot, is it just the defensive skills of my SF guarding him that affect his shot?
You need to read the game narrative to see who is guarding whom during the game. In MtM, this is NOT always SF on SF because the game is realistic and has switches and defensive help. This also happens in all forms of zone. So the other guy team will not always be guarded by his opposite number, to say nothing of the times your backups will be in the game.
Or does my overall defense ability affect him?
If by 'overall defensive ability' you mean your team ratings, you need to understand those team ratings are NEVER used by the game engine. The team ratings are solely informational and calculated according to the formula in the Game Manual. The boost or decrement to ratings described in the Game Manual are applied to the individual ratings during the play of the game.