Finally got a chain (/wheel) tensioner. As my dropouts are home made I had to do some dremeling on the tensioner to make it even work. But being 3 bucks instead of 5 + shipping online not a big deal. Dremeling worked great and now my chain won’t slide at all when I bike, about fucking time.