The problem in your code arises because b^{4} is not the same as b**4. If you do print(tree(ex)) on the output, you can see that one of the terms has b^{4} and the other b**4 (represented as \pow{b}{4}.
You can solve this in various ways. One way is to convert the b**4 to b^{4} with another rule. But since you really mean a power, perhaps a quicker way to achieve the wanted result is to use powers everywhere, not superscripts:
a::Symbol;
b::Symbol;
{a**m?, b**n?, a, b}::NonCommuting;
{m,n,r,s}::Integer.
r1 := {
a b**m? -> (1+b a) b**{m?-1},
a b -> 1 + b a,
};
def act(ex):
converge(ex):
substitute(ex,r1)
distribute(ex)
collect_factors(ex)
return ex
act($a b**5$);
I have cut out a few other unnecessary steps in your act function (you don't need to call collect_terms to collect terms in an exponent, and you also never need to call the brute force hammer simplify unless you need a simplification for which SymPy is needed).