The Potential and Limitations of OpenAI's Custom GPTs.

Despite the OpenAI saga in late 2023, the GPT App Store will officially launch next week this January. I'm a bit excited; I created two "custom GPT"s, and building them was refreshingly easy, though limited in capability.

OpenAI actually tried launching the App Store sooner, but I suspect the Sam Altman ousting affected the product timeline:

Initial experience with custom GPTs

This was a good decision- the GPT builder was a bit clunky. The "Update" button sometimes wouldn't work and updating the avatar image sometimes wouldn't save, but these are minor nits. My main issues are with the bot builder and downloadable files.

Firstly, building a chatbot through the same chat interface is incredible. Very futuristic, or 'Westworldian'. It's not entirely magic though; the builder summarizes how you want the chatbot to behave into a body of text.

Building the chatbot via chat updates the 'Instructions' text body

This concept is fancily worded as "prompt engineering", or providing context to a large language model like ChatGPT for better, more insightful answers.

[Without Prompt]
- What programming language should I learn as a beginner?
[With Prompt]
- Pretend you're a principal software engineer. Be opinionated and direct. Limit responses to three sentences. Always end answers with a follow up question.
- What programming language should I learn as a beginner?

Bigger issues I ran into building a custom GPT chatbot

Instructions are constantly overwritten.

At some point, the changes I instruct to the chatbot overwrite or rephrase previously crafted instructions, some of which I spent considerable time manually writing out. This can easily be solved with a 'revert changes' feature; if you don't like the new prompt, go back to the previously saved one.

Uploaded knowledge files were initially downloadable.

When I first built chatbot Tim, an iOS interview coach, I noticed the PDF files I uploaded were downloadable by just asking Tim for them. Well that's no fun. How can chatbots differentiate themselves on the GPT store if content can be stolen, repurposed and used by others?

Fortunately OpenAI acknowledged this when they initially delayed the GPT store launch.

There have also been questions around uploaded files. Uploaded files are downloadable when using Code Interpreter so we’ve made this feature default off and added messaging to better explain this.

OpenAI has indeed disabled file downloading since- but interestingly enough, you can still pry and ask for file details: "Where is this information from?"... "What's the name of the file?" ... "What else is written in the document"?

An example with a publicly accessible interview prep doc provided by Meta.

I planned to build a unique knowledge base by combining original content with interview prep docs from places like Meta. But, from what I've seen, if you're making a GPT, you should pretty much assume any doc you upload is up for grabs. Basically, anyone can get to its contents just by asking the right questions.

While using uploaded files vastly lowers the barrier to publishing a chatbot, the uploaded content is way less defensible than if you had integrated GPT Actions / data retrieved from your own API.

Then... who are custom GPTs for?

I don't know. Maybe AllTrails since it's featured on the GPT home page 😆?

As of January 2024, with the GPT store having been introduced, there are many featured GPTs from independent authors— maybe because API-powered competition hasn't made its way in yet. I sound like a grumpy skeptic but I don't wish to. I don't even think a GPT's base capabilities and file uploads can sustain an edge over the default GPT-4 chatbot, let alone an API/Action powered one.

Given the 512 MB limit per file upload, it's conceivable that someone could upload a highly concentrated knowledge base and succeed. But for the average GPT author, achieving indie success with file uploads alone appears unlikely.