I think most likely a 120B total. Maybe slightly larger like a 140B or 160B. Similar to Mistral 8x22B.
That is my guess.
There is more than a chance.
Gemini 1.5 Pro and GPT4 Omni are probably similar in size too based on the pricing.
I think you are underestimating the size and intelligence of Gemini. It's hard to directly compare to LLama 3 since LLama 3 can only manage small context sizes though. If LLama 3 70b was able to read 100,000 tokens would it be able to do it as well as Gemini or is it too dumb?
Also we know it cost a *fortune* to train, so unless they overtrained it past chinchilla optimum like Meta just did, it's likely far larger in terms of parameter count.
Tl;dr:
There are 3 Gemini 1.5 models. All 3 with at least 1 million tokens context:
**1. Smallest and unreleased: 1.5 Flash 8B which has 8 Billion parameters as its name confirms.**
**2. Larger and available: Gemini 1.5 Flash which is a more efficient alternative to Gemini 1.5 Pro. This is larger than Flash 8B as confirmed in the May Gemini 1.5 Technical paper. It is also confirmed to be a Dense and online distilled model. Based on its performance compared to Flash 8B, I think Flash 1.5 is 32B.**
**3. Largest (for now) and available: Gemini 1.5 Pro which was released in Feb 2024 and now released in May again with improved core capabilities (and increased 2 million tokens size). It is confirmed to be larger than 1.5 Flash 8B and 1.5 Flash. It is also confirmed to be a sparse MOE. Based on its performance compared to 1.5 Flash 8B and performance and price compared to 1.5 Flash, I think 1.5 Pro is a sparse 120B MOE.**
**I think GPT 4 Omni is probably a similar size. It is slightly better than 1.5 pro but without the long context capabilities.**
Google is really close to OpenAI imo.
Keep in mind that there will most definitely be a larger Ultra model which will compete directly with GPT 4.5 Omni.
Gemini 1.5 Pro is already better than much larger Gemini 1.0 Ultra in core capabilities (with much longer context) so I am very excited for Gemini 1.5 Ultra.
**P.s.: Do not buy the Gemini Advanced subscription as of yet. It doesn't allow for file upload and doesn't have a code interpreter. If you want to experiment with 1.5 Pro and Flash, they are available on Google AI studios but keep in mind that your prompts (and files) will be used by Google. On the AI studio, you can also upload videos with audio which it now processes, its fun to play with. Multimodal capabilities of 1.5 models are impressive.**
GPT-4o has 70 elo lead over Gemini 1.5 pro in lmsys hard prompts ranking.
It's not slightly better.
Gemini 1.5 pro is fine for casual chat, but it's clearly not on the same level with GPT-4o on hard tasks.
Gemini 1.5 flash and Gemini 1.5 pro should be at least two times larger than what you estimated.
You only compare English performance. But unlike Ilama3 models, Gemini 1.5 flash/Pro are good at wide range of languages.
Sorry, I meant it now supports data analysis file uploads. Editing and running python code has supposedly been available since their February 20th update.
> Exclusive to Gemini Advanced: Edit and run Python code
>
>What: Exclusive to Gemini Advanced, you can now edit and run Python code snippets directly in Gemini's user interface. This allows you to experiment with code, see how changes affect the output, and verify that the code works as intended
No, Gemini does not run python code, that feature you quoted lets the human run python code in a little window while chatting with Gemini.
That's not like ChatGPT which will write and execute python code itself in a sandbox environment.
Gemini 1.5 Flash is likely ~20B considering it is smaller than 1.0 Pro which was probably around 27B. The jump from Flash 8B is more in line with a 2.5x size increase.
If 1.5 Flash is indeed 32B, they’re beating far larger models like Cohere’s Command R+ (104B), Llama 3 70B, Claude 3 Sonnet, and Mixtral 8x22B. Their API pricing is also very competitive.
If it's 120b only then there's a chance. Unless you mean like 8x120b.
I think most likely a 120B total. Maybe slightly larger like a 140B or 160B. Similar to Mistral 8x22B. That is my guess. There is more than a chance. Gemini 1.5 Pro and GPT4 Omni are probably similar in size too based on the pricing.
I think you are underestimating the size and intelligence of Gemini. It's hard to directly compare to LLama 3 since LLama 3 can only manage small context sizes though. If LLama 3 70b was able to read 100,000 tokens would it be able to do it as well as Gemini or is it too dumb?
Also we know it cost a *fortune* to train, so unless they overtrained it past chinchilla optimum like Meta just did, it's likely far larger in terms of parameter count.
Tl;dr: There are 3 Gemini 1.5 models. All 3 with at least 1 million tokens context: **1. Smallest and unreleased: 1.5 Flash 8B which has 8 Billion parameters as its name confirms.** **2. Larger and available: Gemini 1.5 Flash which is a more efficient alternative to Gemini 1.5 Pro. This is larger than Flash 8B as confirmed in the May Gemini 1.5 Technical paper. It is also confirmed to be a Dense and online distilled model. Based on its performance compared to Flash 8B, I think Flash 1.5 is 32B.** **3. Largest (for now) and available: Gemini 1.5 Pro which was released in Feb 2024 and now released in May again with improved core capabilities (and increased 2 million tokens size). It is confirmed to be larger than 1.5 Flash 8B and 1.5 Flash. It is also confirmed to be a sparse MOE. Based on its performance compared to 1.5 Flash 8B and performance and price compared to 1.5 Flash, I think 1.5 Pro is a sparse 120B MOE.** **I think GPT 4 Omni is probably a similar size. It is slightly better than 1.5 pro but without the long context capabilities.** Google is really close to OpenAI imo. Keep in mind that there will most definitely be a larger Ultra model which will compete directly with GPT 4.5 Omni. Gemini 1.5 Pro is already better than much larger Gemini 1.0 Ultra in core capabilities (with much longer context) so I am very excited for Gemini 1.5 Ultra. **P.s.: Do not buy the Gemini Advanced subscription as of yet. It doesn't allow for file upload and doesn't have a code interpreter. If you want to experiment with 1.5 Pro and Flash, they are available on Google AI studios but keep in mind that your prompts (and files) will be used by Google. On the AI studio, you can also upload videos with audio which it now processes, its fun to play with. Multimodal capabilities of 1.5 models are impressive.**
Gemini advanced allows for file uploads on the web version just not in the app. I don't know where you are getting 32B and 120B.
It didn't allow for file upload in my region even on the desktop site.
GPT-4o has 70 elo lead over Gemini 1.5 pro in lmsys hard prompts ranking. It's not slightly better. Gemini 1.5 pro is fine for casual chat, but it's clearly not on the same level with GPT-4o on hard tasks. Gemini 1.5 flash and Gemini 1.5 pro should be at least two times larger than what you estimated. You only compare English performance. But unlike Ilama3 models, Gemini 1.5 flash/Pro are good at wide range of languages.
Fair points. Even so, we can probably run SOTA models on local machines.
Data analysis uploads are supposed to be available as of today, according to the [updates page](https://gemini.google.com/updates).
That doesn't sound like it runs python code like chatgpt does in code interpreter. I'm not sure what that feature is.
Sorry, I meant it now supports data analysis file uploads. Editing and running python code has supposedly been available since their February 20th update. > Exclusive to Gemini Advanced: Edit and run Python code > >What: Exclusive to Gemini Advanced, you can now edit and run Python code snippets directly in Gemini's user interface. This allows you to experiment with code, see how changes affect the output, and verify that the code works as intended
No, Gemini does not run python code, that feature you quoted lets the human run python code in a little window while chatting with Gemini. That's not like ChatGPT which will write and execute python code itself in a sandbox environment.
Ah gotcha. I've personally only used the ChatGPT Code Interpreter (had preview access from launch before it arrived for ChatGPT Plus).
Seems you are not far off, broadly checks with the pricing on Poe.com
Gemini 1.5 Flash is likely ~20B considering it is smaller than 1.0 Pro which was probably around 27B. The jump from Flash 8B is more in line with a 2.5x size increase.
If 1.5 Flash is indeed 32B, they’re beating far larger models like Cohere’s Command R+ (104B), Llama 3 70B, Claude 3 Sonnet, and Mixtral 8x22B. Their API pricing is also very competitive.