NVIDIA Launches Compact AI Model Mistral-NeMo-Minitron 8B

NVIDIA has introduced the Mistral-NeMo-Minitron 8B, a compact yet powerful language model designed to deliver state-of-the-art accuracy while being efficient enough to run across GPU-accelerated data centers, clouds, and workstations. This miniaturized version of the Mistral NeMo 12B model, released in collaboration with Mistral AI, represents a breakthrough in balancing model size with performance.

Balancing Size and Accuracy:

Developers of generative AI often face the challenge of choosing between a larger, more accurate model and a smaller, more efficient one. The Mistral-NeMo-Minitron 8B resolves this tradeoff by offering the best of both worlds. This model is optimized to run on NVIDIA RTX-powered workstations, making it accessible to organizations with limited resources while maintaining exceptional performance across various benchmarks.

“We combined two AI optimization methods—pruning to shrink the Mistral NeMo’s 12 billion parameters to 8 billion and distillation to improve accuracy,” said Bryan Catanzaro, vice president of applied deep learning research at NVIDIA. “By doing so, Mistral-NeMo-Minitron 8B delivers comparable accuracy to the original model at lower computational cost.”

Efficiency and Flexibility:

The compact nature of the Mistral-NeMo-Minitron 8B enables real-time performance on workstations and laptops, making it easier for organizations to deploy generative AI capabilities efficiently. Running models locally on edge devices also enhances security by keeping data on the device, reducing the need for data transmission to servers.

Developers can access the Mistral-NeMo-Minitron 8B as an NVIDIA NIM microservice with a standard API or download it directly from Hugging Face. Additionally, an upcoming NVIDIA NIM microservice, deployable on any GPU-accelerated system, will provide further flexibility for developers.

Leading Performance in Its Class:

For an 8-billion-parameter model, Mistral-NeMo-Minitron 8B excels in various AI tasks, leading across nine popular benchmarks that assess language understanding, common sense reasoning, mathematical reasoning, summarization, coding, and more. Optimized for low latency and high throughput, the model delivers faster user responses and increased computational efficiency in production environments.

For those requiring an even smaller model for applications like smartphones or embedded devices, developers can use NVIDIA AI Foundry to further prune and distill the Mistral-NeMo-Minitron 8B into a more compact neural network tailored to specific needs. NVIDIA AI Foundry provides a comprehensive solution, including foundation models, the NeMo platform, and access to NVIDIA DGX Cloud and AI Enterprise.

Advanced Techniques for Optimal Performance:

To achieve the high accuracy of the Mistral-NeMo-Minitron 8B, NVIDIA employed a combination of pruning and distillation. Pruning removes less significant model weights, reducing the size of the neural network, while distillation retrains the pruned model to restore and even improve accuracy. This approach significantly reduces the compute cost, requiring up to 40 times less computational power compared to training a smaller model from scratch. You can read the NVIDIA Technical blog for more details.

NVIDIA also introduced Nemotron-Mini-4B-Instruct, another small language model optimized for low memory usage and fast response times on NVIDIA GeForce RTX AI PCs and laptops. This model is available as part of NVIDIA ACE, a suite of generative AI-powered digital human technologies.

Both models can be experienced as NIM microservices through a browser or API at ai.nvidia.com.