Nehemiah is speaking in this verse. The context preceding this verse is the poor Jews coming to Nehemiah, complaining about their being taken advantage of by their countrymen of means. They've had to borrow money from the rich, sell their fields, and even pawn their children to the wealthy to stay alive themselves. You can picture Nehemiah hearing this, and calling together the leaders of the people to chastise them. It doesn't even say in the context that he paused to pray. I can see him with hands on hips, palms out, indignantly calling out those exploiting the poor. He says that they just got out of enslavement to another nation. Need they do it to each other?
For his part, Nehemiah also leads by example. Although a leader, he doesn't use his position to his advantage. He even feeds a considerable number of people out of his own pocket. He could pursue land acquisition for himself, but he refuses to be distracted by material gain, in order to focus on his mission -- rebuilding the wall. He has his sleeves rolled up, and expects his cabinet to do so as well.
Nehemiah avoids the subtle allure of secular, material stuff of earth. This stuff can be a distraction to us also. It isn't that the stuff is so bad, but the distraction from true purposeful living is the problem. Our energy only goes so far, and property or other wealth is to be a tool as well as an enjoyment, but shouldn't be a siphon of our energy in the wall building God has set us to do.
Nehemiah and the wall