Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling | #llms #training #syntheticdata #rephrasing #wrap #generativeai

Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling

Large language models are trained on massive scrapes of the web, which are often unstructured, noisy, and poorly phrased. Current scaling laws show that learning from such data requires an abundance of both compute and data, which grows with the...

"Are you still using real data to train your AI? Nvidia's Rev Lebaredian says synthetic data can make AI systems better and maybe even more ethical." Of course, because the simulation that generates the synthetic data knows the 'ground truth'.

"For example, in autonomous vehicles you want your car to know, by looking through its sensors at the world, the relative 3D positions of all of the cars and pedestrians around it. But it's just getting a 2D image that's nothing but pixels; there's no information about it. So if you're going to train a network to infer that 3D information, you first have to draw a box around things in 2D and then you have to tell it, 'Here's how far away it is based on the particular lens that was used with that sensor.' But if we synthesize the data in Omniverse, we have all of that 3D information at full physical accuracy. We can provide exact labeling without the errors that a human would introduce into the system. So the resulting neural network that we train is going to be smarter and more accurate."

"Are you still using real data to train your AI?

#solidstatelife #ai #computervision #syntheticdata

Are You Still Using Real Data to Train Your AI?

It may be counterintuitive. But some argue that the key to training AI systems that must work in messy real-world environments, such as self-driving cars and warehouse robots, is not, in fact, real-world data. Instead, some say, synthetic data is...