I want the party to have an exploration/social encounter with the bagman (not Combat) 

I considered having a Bag of Holding sitting unguarded in the middle of a campsite. No one is around, but there are scratch marks leading toward the bag. 

I was thinking of making it a skill challenge with a bag of holding as a reward. 

Has anyone ever done anything like this before?

From what I gather, the bag man is two things:

• insane
• hungry

So I am unsure how a social encounter might play out here. Furthermore, if any of your players use any metagaming in the slightest, they will know that the bagman can come out of any bag of holding, so they will likely leave that bag well alone, or burn it!

I would focus on what the Bagman wants out of this. Does he want food, or to be moved somewhere with richer pickings? Is there another bag of holding which was turned inside-out that he wants turned the right way around?

Remember that your party may adopt a "kill it with fire" approach immediately, so don't count on it staying a social encounter!

If you want it to be social then it might be better to have them seen outside the bag initially then crawl into it. Then if the party goes to look in the bag it speaks to them from some hidden place in the void of the bag.  Once it is in the bag it's safe so combat is less likely.

only one questions and youll have your encounter. why does he kidnap people?