Let’s talk about teaching machines
In the last decade, how we build software has changed. As the use of Artificial Intelligence (AI) becomes mainstream, data has become the new oil. And while Machine Learning has focused on the methods and algorithms to detect patterns in data, the importance of gathering the right data has led to a new paradigm — a data-centric approach to AI. This means we need a fresh perspective on how we teach machines — encompassing understanding what ‘right’ data means, how we source it, how domain knowledge is captured, how we overcome biases, and how we ensure that we are building the right product.
While this article makes the case for a fresh perspective on looking at the “teaching” side of Machine Learning, if you need a quick introduction to ML, check out this article by Sam Drozdov, which is meant for designers, and a great starting point for the learning methods. Another article that’s close in spirit is this one by Joel van Bodegraven, which talks about the design principles of AI-driven UX. Lastly, this article by Punchcut talks about how to design for the data capturing the side of ML UX.
A quick introduction to how AI is being built
Today, AI is used in the products and services we use daily for automating routine tasks, simplifying interactions by assisting users, and providing delight with personalization. The best applications provide a subtle, but important, bump in the user experience that sub-consciously prevents us from switching back to a less intelligent alternative.
Google Mail is a great example of this (I’m sure other mail clients have similar features as well). It provides an awesome User Experience, in part thanks to the incredible front-end and back-end engineering work, and in part by providing intelligent features like the following:
- Automation: I’ve rarely seen spam in my main inbox in the last few years. There’s probably a classifier sitting somewhere in the cloud that has learned patterns that spammy emails using massive email datasets.
- Assistance: Typing emails is easier than ever thanks to the auto-complete functionality. It even knows the context of the email, helping me type out pleasantries with the names of the recipients. It’s learned how to do this by finding patterns in how people compose emails with family, friends, colleagues, etc.
- Personalization: It marks stuff that is important to me and sorts the mail into personal, social updates, promotions, updates, and forums — making it easier for me to know where stuff is. It’s learned from the few examples I’ve given it about what emails I find important.
I use the word “learned” above generously. Google Mail doesn’t have a mind of its own. To create such artificially intelligent features, Data Scientists use Machine Learning algorithms to experiment, create, and iterate models that detect patterns in incoming data and generate insights or perform such knowledge-driven tasks in a human-like way. Product teams then work together to create this experience by engineering the front-end, and the back-end. To keep the model up to date, MLOps teams create the pipelines for sourcing the data, preparing the training examples, and training and deploying the models.
Clearly, the effort needed to get an ML model in production is large.
Why do we need to talk about teaching machines?
In the long term, we’re all going to need to know how to teach machines to do what we need. Think of machine teaching as a way to customize the software to adapt to you. AI, after all, exists to assist us. We’ll need to know when it is working well for us, its shortcomings, how it responds to us training it, and how to fix it when it’s “got a mind of its own” and doesn’t work the way we want it. It’s a skill almost everyone will need — from the biology researcher who wants the intelligent microscope to detect and analyze the specific cells they are looking for, to the farmer who is running intelligent tractors in their fields, to the patient and the doctor who are both relying on the results of the cancer test that was performed using AI. Ok, maybe it’s not as long-term as I mentioned at the top of this paragraph.
The good news is we know how to teach machines already. We’ve been teaching the Netflix algorithm to recommend us movies that we like by rating them. Even teenagers know how to get the right content they are looking for on TikTok by liking and watching what they want to see more of, and quickly scrolling over what’s cheugy.
In the short term, product teams shipping AI features have the most pressing need to make sure that they are using the right data to build their products and services. After all, a model trained on poor data will perform poorly. As part of this new approach, they need to take a purposeful effort to create the right dataset to train the model with, while keeping the model/code constant (until at least there’s no more performance improvement to squeeze out of the data).
This paradigm is called Data-Centric AI.
The focus for data-centric AI should then be on the “teaching” side (as opposed to the “learning” side of ML). It needs:
- The right teachers
- The right coursework
- The right training routines
The Right Teachers
While Data Scientists have been squarely in the driver's seat on requesting the data and creating the models so far, the data-centric approach needs a strong team that understands the nuances of the data, the use cases, AND how the AI reacts to those so that they can teach the machine right. This creates an opportunity for creating new roles with new skills on how to treat the machine that is learning as a black box and teach it effectively.
The product manager should define what the model should do. In only a handful of companies that I have talked to, have I found that a product manager is leading what model needs to be built. Most often, the data scientist defines the model requirements, creates the specifications, and manages the annotation tasks. They’re also far removed from the end customer. If you want the AI features you’re building to see the light of day, a product manager who understands the business impact and the user experience improvement needs to define what the model should do.
A subject matter expert should grade the data. When you don’t control the data, the subject matter expert is brought in as an advisor on the domain. However, for the data-centric approach, the subject matter expert should be the one who provides the initial examples of what good training data looks like, and how well the model is performing on unseen data. While annotation teams and data scientists have a role to play here, in almost all teams I have talked to, the final arbitrator on what is good data is only one or two people.
An operations manager should make sure that the teaching is progressing in the right timeframe. It’s important to get your feature out in the hands of the user so that you get the right feedback on the user experience. This means that while data scientists focus on improving the model, we need someone to focus on making sure we are collecting a variety of data for the different use cases, especially when the model comes back with some areas that need improvement. Assuming that the model training part is (semi) automated, this role makes sure the model is learning well for iterating data.
A data-wholesomeness champion should ensure that the data is representing variety and diversity in the real world. As teachers are human, we are prone to introducing our human biases in the models we build. While listening to different panelists at a healthcare conference last December, I couldn’t help but wonder that if the most cutting edge AI solutions are built by hospitals with affluent patients, the data that the model is based on may lack the examples from lower-income neighborhoods, or an underrepresented racial/ethnic groups. Bias is often unintended, but can also result from a biased group of data annotators and reviewers. Removing bias is essential for creating ethical AI solutions. Fortunately, the movement to rid the training data from bias is gaining momentum.
There are other stakeholders involved as well (e.g. the project sponsor, the data annotators, the annotation project managers, etc.), but perhaps I can elaborate on them later. It really takes a village.
The Right Coursework
You might have seen in the section above, that the role of the teachers is to create the right “coursework” for the model. Think data management.
Generally speaking, to make sure that the model performs well in the real world, the data that it is trained on needs to match that. This puts some constraints on what data can we train the model with and how it is sourced. The best data is sourced from a situated implementation where the end AI feature is going to be deployed. For example, for our Google Mail case, it was the emails themselves. For a model deployed inside a factory, it might be a camera mounted on top of the conveyor belt. For autonomous driving, it’s real streets in different weather situations.
The right data also depends on the type of model we are building. That is, what are we teaching the model to do. When teaching machines to see (i.e. computer vision) the capture device and lighting conditions need to match the deployment conditions. When teaching machines to read (i.e. natural language processing) we need to understand that data is going to come with noise — so we might need to automatically clean up data while training the model as well as deployment. There are similar considerations when teaching machines to personalize (i.e. recommendation), to listen (i.e. audio processing), etc.
Knowing when the coursework needs to change and improve is also important. Data may drift after deploying models to production. Concept mappings may also shift depending on changing user behavior.
The Right Training Routines
While the right teachers can ensure that the coursework is right, the right processes are needed to make sure that the model is learning well and is bringing the intended UX.
Data sourcing and annotation can get expensive really fast. There are tools available for Data Scientists that help pick the subset of data to annotate the model with, assisting labelers while annotating, detecting data drift, and identifying biases (hyperlinks are just examples, and there are other tools out there as well). We need more tools to help the teachers above teach better without having to understand the complexities at the level of data scientists. This is an area where no-code solutions, such as AI Hero, can shine.
Once you get a model trained, the teaching plan doesn’t stop there. Instead, it’s an iterative process where you perform routine updates depending on the model performance. For example, you might find that some data that is similar, is actually labeled differently. Or perhaps you identify inconsistencies in the annotations and need to change the annotation instructions, and as a result re-annotate some older data.
Ontology update and management also fall into this bucket — when you decide that you need a new class or attribute in the class and decide to request partial annotations for the updated class or attribute.
If from the article so far, you have started to think of your AI as a little bot that needs to learn and score great in the real world, let me burst your bubble, and show you what the classroom really looks like. Instead of having only one AI model, think of different model versions as progressing through the classroom. You’ll always have a “champion model”. As your teachers collect more data, clean more data, and iterate on it, your data science team can produce one or more contender models that need to score better than that model to become the new champion. If the right data means changing the tests on newly discovered use-cases, then perhaps an older model might perform better and become the new champion. You need a strong MLOps platform to support this data-centric AI process.
I am perhaps only scratching the surface here with what actually goes into creating models that work in the real world. I’ll be writing a series of blog posts that help dig down deeper. If you’re interested in learning more, follow me for more articles like these. If you’re interested in sharing your thoughts and perspectives I’d love to talk. By focusing on how we teach machines aside from Machine Learning, I’m hoping we can speed up the deployment of AI features that help the human experience. Until next time.
If you like the article please follow me on Medium, whose partner program changes are affecting writers like me.
