All I could find searching for this poem by author and title were posts by people saying they couldn't find it. And when I had found it out in the real world, and Googled one of the lines not quoted in Chesterton's Autobiography, I got no hits at all.
The Rebel, Hilaire Belloc
From 'Sonnets and Verse', first published 1923, 1954 edition.
There is a wall of which the stones
Are lies and bribes and dead men's bones.
And wrongfully this evil wall
Denies what all men made for all,
And shamelessly this wall surrounds
Our homesteads and our native grounds.
But I will gather and I will ride,
And I will summon a countryside,
And many a man shall hear my halloa
Who never had thought the horn to follow ;
And many a man shall ride with me
Who never had thought on earth to see
High Justice in her armoury.
When we find them where they stand,
A mile of men on either hand,
I mean to charge from right away
And force the flanks of their array,
And press them inward from the plains,
And drive them clamouring down the lanes,
And gallop and harry and have them down,
And carry the gates and hold the town.
Then shall I rest me from my ride
With my great anger satisfied.
Only, before I eat and drink,
When I have killed them all, I think
That I wll batter their carven names,
And slit the pictures in their frames,
And burn for scent their cedar door,
And melt the gold their women wore,
And hack their horses at the knees,
And hew to death their timber trees,
And plough their gardens deep and through-
And all these things I mean to do
For fear perhaps my little son
Should break his hands, as I have done.