The last few weeks have been very interesting in the AI space. We saw the release of a new updated version of ChatGPT from GPT-3.5 to GPT-4. Within a couple of days, Google soft-launched their competitor AI chatbot, Bard (available in the US and UK). Adobe dropped Firefly, Canva dropped their own image AI, and Bing added DALL-E.
We also saw Opera browser incorporate AI prompts using GPT and DuckDuckGo using it to summarise search queries. But by far the biggest development was the release of ChatGPT plugins.
Plugins are big for two obvious reasons. Number one, now that ChatGPT has access to third-party tools it can give more accurate answers from trusted sources and databases. For example, you may ask the GPT to solve a math equation, and now it can use Wolframa to provide an answer. And secondly, plugins are big because now GPT can automate tasks across multiple tools — it’s not just responding, it can now create actions.
It’s going to be awesome when more plugin developers get their hands on the documentation. We’ll see some really cool automation paths in the coming weeks.
I want to discuss a few ways that developers can and have been using these plugins but first let’s talk about the actual plugins so you can get an idea of what’s available at the moment.
According to the OpenAI blog, there are only 12 plugins currently available to a small set of paid users. After an Alpha period, more developers will get access and API users will be able to integrate plugins into their products. Here are the available third-party ChatGPT plugins:
Expedia: Flights, cars, hotels, and travel
FiscalNote: Real-time data sets for legal, political, and regulatory data
Instacart: Order groceries from your local grocery stores
KAYAK: Flights, cars, hotels, and travel
Klarna Shopping: Search and compare prices from online shops
Milo Family AI: Co-pilot for parents, powered by GPT-4
OpenTable: Restaurants and restaurant reservations
Shop: Search for millions of products from the world’s greatest brands
Speak: Learn how to say anything in another language
Wolfram: Access computation, math, curated knowledge & real-time data
Zapier: Workflow automation across 5000+ apps
The best NEW plugins
To be honest, all of these plugins will be very useful to many people. Some people have been comparing it to an app store. The potential is crazy. But I only want to focus on a few of these plugins I think are the most interesting.
1. Browsing has been added
Up until now, ChatGPT-4 hasn’t had access to live data, for example:
With the browsing plugin, we can get up-to-date information. Your research will be current and you’ll be able to ask more specific questions. The plugin also lets you see exactly the path the chatbot took to arrive at the answer. It’ll display a list of what was searched and the pages it read to arrive at its final answer. This allows you to fact-check the bot and source data.
I’ve also seen it used for solving code issues. It’s reassuring to know that the bot is pulling information from StackOverflow or relevant documentation to solve your queries.
2. Code interpreter
ChatGPT has so far been a very impressive coder. It can write and read in multiple languages. It can QA code, or provide constructive feedback. But if you’ve used it for coding, you’re aware that a lot of the output has to be put into a different environment to test it. With the Code Interpreter (CI) plugin you can now do much more of the development inside ChatGPT.
I would recommend reading ChatGPT + Code Interpreter = Magicby Andrew Mayne. He goes into depth on the CI plugin and shows its various uses. If you don’t have time I’ll give you the TL;DR.
Solving mathematical problems, both quantitative and qualitative
Doing data analysis and visualisation
Converting files between formats
ChatGPT can provide you with a downloadable link tailored to the file type you ask it for. Sound file types and .txt were demonstrated. Not only do you see the code snippet but you’ll also see the result.
In the article I mentioned above, Andrew creates mini-games, sound files, working QR codes, face-finding algorithms, images converted to ASCII images, calendar invites, drawings, and an animated graph.
Retrieval is like the admin plugin. The retrieval plugin enables ChatGPT access to personal or organisational information sources. With access to data sources, files, notes, emails, and documentation, you’ll be able to query the bot for any task. For example, a HR person might want to know an employee's tax details. They query GPT and it gives them the number. They could even ask for a list of all employee tax IDs, turn the list into a table and create a presentation all within the same environment.
ChatGPT can be a data management tool. The tedious task of creating a report is going to be almost eliminated. Think about it, you ask for a list of data (maybe it’s linked with Power BI or Google Analytics) without having to create a ticket for the data team. You query the bot about the data and then turn it into a presentation or animated graph. What could have taken a week and crossed teams, takes you five minutes?
Indexing and searching documents
Summarising email and draft response
Searching for lost or missing files
This is where it gets fun! Using the browsing or retrieval plugin and the Zapier plugin you can set up automation sequences to trigger actions. So from within the same environment you can search for data, turn that information into a report or image and send it off to one of your email contacts.
That’s just a simple example, I’m sure you can think of a bunch of tasks within your role or within a department that can be completely automated with the integration of ChatGPT.
Search Notion database for any item and add or update it directly inside Notion
These are the example provided by Zapier, and to be honest I don’t think these are great examples. It’s easy enough to open slack and send a message — it doesn’t really make sense to set up Zapier for that, especially when it’s a paid app. Generating weekly reports or data retrieval tasks that are more or less predictable events is the first thing that comes to my mind.
Now this plugin isn’t available in the store it was created by a Beta user. He created a Swift Playground plugin so that he could quickly create iOS applications. In the example below he is creating animal flashcards.
👾 ChatGPT Plugin for 🍏 Swift Playgrounds
"Let's make a flashcards app for emoji and their unicode names"
This next user (with no Swift experience) creates a mostly functional iOS app in less than two hours with payment and OpenAI integration. The idea for the app is you tell it what’s in your fridge and your fitness goals and it tells you what you should eat (and can order from Instacart). Very impressive.
RIP IOS Developers 💀
I just built a 95% functional IOS app in less than 2 hours with GPT-4. (With Payments and OpenAI integration)
I have zero Swift experience (I hired a freelancer to pair program thou.)
There are a bunch of secret plugins not yet accessible. A user was hacking the ChatGPT API and found over 80 hidden plugins. For a few hours, it was possible to use these plugins but OpenAI was quick to fix the big. Thankfully there are some screenshots of the hidden plugins so we know what’s coming.
This morning I was hacking the new ChatGPT API and found something super interesting: there are over 80 secret plugins that can be revealed by removing a specific parameter from an API call.
Okay, so nothing to hold your breath over, but still cool!
Response to the launch of plugins
Overall, plugins are a game-changer and have just added a lot more value to a product that is already groundbreaking and changing how we work in real-time. Let’s just wait and see what developers out there can do with this technology once it’s opened to the public. To leave you, here’s Brock asking ChatGPT to summarise the public response to the new plugin update.
The rich text element allows you to create and format headings, paragraphs, blockquotes, images, and video all in one place instead of having to add and format them individually. Just double-click and easily create content.
Static and dynamic content editing
A rich text element can be used with static or dynamic content. For static content, just drop it into any page and begin editing. For dynamic content, add a rich text field to any collection and then connect a rich text element to that field in the settings panel. Voila!
How to customize formatting for each rich text
Headings, paragraphs, blockquotes, figures, images, and figure captions can all be styled after a class is added to the rich text element using the "When inside of" nested selector system.