@beastMan3b@VarunGangal There is a 10-15% improvement for end to end training time for 100m parameter models and this improvement increases with model size. In the paper for instance we benchmarked that the feed forward part of transformers would have a speedup of more than 60% at bigger sizes.