Jelena - Apologies for such a late response, and maybe you've already hosted one and have your own tips, but I wanted to share some thoughts...
First, I think that a clothing swap party for your apartment building sounds like a great idea. It's easy for people to get together, and will be a nice 'excuse' to get to know neighbors. I could see this turning into an annual/regular event with rotating hosts.
I think that getting a few people to play the role of hosting/organizing together may be helpful, so that not all the responsibilities are on the host. For example, the person who's offering the space prepares some snacks, other co-hosts help get the word out, etc. And if you do turn it into a regular thing, someone from the co-host group can be the "backup" when no one else in the building can host.
Depending on the size of the party you want to have, I think that encouraging participants to invite their friends can be good because - (1) more the merrier, and (2) more diversity in people joining the party = more likely to have different styles of clothes, which will make the swap more exciting, I think.
Also, as a host/hosts, you can remind everyone to relax, have fun, and enjoy clothes-swapping in the way they want to. Some people may find lots that fit, some may not (which is OK, as you can still enjoy the company), and some may bring lots of great dresses and some may bring mostly casual stuff (which is great, so the party will have a great mix of styles to choose from). Some people may pick everything they like, some of which they end of not liking and bringing to another clothing swap party in a few weeks. I think all these are OK, as long as everyone has fun and the basic rules that the group decides on (e.g. clothes need to be clean and wearable) are respected.
Some of the clothing swap parties I've been to also allowed jewelry and accessories. Maybe some people find it easier to bring things that way.
Lastly - something to consider for what happens after everyone has picked their clothes: you can, as a group, find a local charity to donate whatever remains after the party. Or ask everyone to take home what they brought that wasn't picked by others (and offer tips on recycling, upcycling, etc.). I also think it's a good idea to ask everyone to bring their own reusable bags, boxes, etc.