In the previous year, OpenAI unveiled its most powerful large language model (LLM), GPT-4, marking a significant leap forward from its predecessor, GPT-3.5. On its release, OpenAI showcased the model’s capabilities, demonstrating impressive performance across various assessments. GPT-4 achieved noteworthy percentiles in exams such as LSAT, SAT, Math, GRE Quantitative, and GRE verbal and writing. The recent reports also suggested that OpenAI extensively trained the LLM using data from millions of hours of YouTube videos. Now, under the leadership of Sam Altman, OpenAI has announced another upgrade for GPT-4.
1. Expanding Horizons: GPT-4 Turbo with Vision
In the latest update, GPT-4 has been provided with the ability to process image inputs and demonstrate improved problem-solving capabilities. OpenAI explained in a blog post that GPT-4, as a large multimodal model, excels in tackling complex problems with higher precision compared to its predecessors, owning to its expanded general knowledge and advanced reasoning abilities.
2. GPT-4 Turbo Integration
The most recent version, termed GPT-4 Turbo with Vision, is now widely accessible as an API for developers. OpenAI has also suggested that GPT-4 Turbo with Vision will soon be integrated into ChatGPT, although specific details have not been disclosed yet.
In a post on X, OpenAI announced, “GPT-4 Turbo with Vision is now accessible through the API. Additionally, Vision requests can utilize JSON mode and function calling.”
By incorporating vision technology, GPT-4 Turbo gains the capability to analyze images, videos, and other multimedia inputs, offering comprehensive responses and insights. This integration of computer vision paves the way for developers to explore a multitude of opportunities, facilitating the development of innovative applications across diverse sectors.
A wonderful aspect of this update is the inclusion of JSON mode and function calling, enabling developers to automate tasks within their applications through JSON code snippets. This advancement holds the potential to simplify processes and boost productivity, facilitating the smooth integration of GPT-4 Turbo with Vision into various projects.
The enhanced AI model features a context window of 128,000 tokens and has been trained on data up until December 2023.
In a related development, a recent article in The New York Times highlighted OpenAI’s struggle with a shortage of training data while developing its Whisper audio transcription model. To address this challenge, the company apparently transcribed over a million hours of YouTube videos to train its GPT-4 language model, despite the legal uncertainties associated with this approach.
OpenAI President Greg Brockman allegedly played a direct role in sourcing these videos. The article also noted that by 2021, OpenAI had exhausted its conventional data sources, prompting discussions about transcribing YouTube videos, podcasts, and audiobooks. Previously, the company had trained its models using a variety of datasets, including computer code from GitHub and educational material from Quizlet.
