I was one of 30 software engineers selected to meet with Open.ai, CEO Sam Altman and Shopify, CEO, Tobi Lütke for an intimate roundtable discussion hosted by Elevate Toronto on May 15 at the Design Exchange. The conversation touched on everything from the potential downsides of AI to why Canada is the right place to develop and commercialize AI technologies that can benefit everyone.
This was the first stop on Altman’s tour—and the first event before he was set to testify before Congress about OpenAI and AI in general. It was the closest thing to a “Willy Wonka” moment I’ve ever experienced, and I was able to ask questions and hear the questions and concerns of other developers in the community.
Altman told the group he was there to listen and answer as best he could. He didn’t want anyone to sugarcoat their questions or concerns. He told us that if we had complaints, now was the time to share them.
After the first question, the conversation started rolling. Altman's answers were honest and straightforward. I’ve only seen him speak online, but he made everyone comfortable asking questions. There were a ton of great questions asked. Many of them are questions our team has been digging into as we work on concepts for ChatGPT-powered solutions for our clients.
My takeaway from the day is that while AI is here—and here to stay—there is still so much work to be done from an engineering and legal perspective before AI ‘takes over the world.’
Getting out of the bubble
While waiting to enter the meeting room, the group discussed why Altman was doing this tour. My thought was that he wanted to get out of the “SF bubble” and hear from developers and leaders with different views on AI. At the roundtable, someone asked that question, and his answer was precisely that—he wanted to get out of the bubble and hear from developers around the world who are using the technology.
Challenges with data residency and regulations
A developer from a professional services firm brought up his concern about data residency. Many countries, Canada included, require customer data to be stored in data centers located in-country. He said that data residency is one of the most significant challenges for his organization working with tools from companies like OpenAI.
Residency isn’t the only challenge. Last week at Google I/O, the search leader launched a new version of Bard, its large language model (LLM) platform, to the world. Well, most of the world. Due to regulatory concerns with their respective governments, developers and users in Canada and the European Union don’t currently have access.
Context is key
My question for Altman was on the platform’s context window. To put that in, well, context, ChatGPT is able to “remember” what a user has asked it during a session. This memory of questions and answers is sent back and forth between the user and OpenAI’s systems as tokens. Each token represents roughly three-fourths of a word. In ChatGPT 3.5, the maximum size of a conversation’s history was 4,000 tokens. That history was expanded to 8,000 tokens in ChatGPT 4. There is even a model that supports 32,000 tokens.
I asked Altman about expanding access to the 32k token model. He replied that the company is resource constrained and they have higher priorities, such as opening up access to more users for ChatGPT 4.
That doesn’t mean public access to the 32k model isn’t coming. Altman said they’d get there eventually—and he hopes to one day get to a 10 million token context window.
Why token and context window matters
One of the most significant use cases talked about with ChatGPT is powering intelligent chatbots that can replace human agents in tasks from customer support to reviewing legal documents. The problem today is that the existing LLMs are trained on large volumes of data, but it doesn’t include business or industry-specific content.
Let’s say you wanted to build a support chatbot for your business that includes being able to answer questions on thousands of pages of product documentation. The ChatGPT LLM doesn’t have access to that data. For each question a customer asks, you first need to include the relevant information in the context window. But with thousands of pages of product documentation, you first need to find the appropriate documentation to include along with the customer’s question.
This could be as simple as asking the customer what model appliance they are requesting help with and then sending that documentation in the context window to ChatGPT. But for instances where there isn’t a simple way to narrow down what documentation to include, developers need to create workflows to help determine what data to include.
Many difficult engineering solutions are still needed to solve this problem. Having access to something like the 10 million token context windows would help. A 10 million token context window would enable us to include massive amounts of data instead of having to sift through it and guess what we think would be relevant.
Why Canada?
Canadian researchers and developers have pioneered AI and machine learning research. But even with the advancements made here, we still struggle to commercialize that research and build businesses around it. During the fireside chat, Lütke said that Canada has a culture of going for bronze and that with AI, let’s do something different.
The roundtable and fireside chat was a fantastic opportunity to not only hear what Altman and OpenAI are doing but also to hear firsthand what other developers and businesses across Toronto and Waterloo are building with tools like DALL-E and ChatGPT. We're actively building solutions using ChatGPT for customers, contact us today to learn how AI can deliver amazing experiences to solve your customers’ problems.
