These graduate students had no idea their teaching assistant was a robot

There are already scheduling robots that are virtually indistinguishable from humans, and recently students at the Georgia Institute of Technology learned that "Jill Watson" -- a teaching assistant they had relied upon all semester -- was in fact artificially intelligent.

“The world is full of online classes, and they’re plagued with low retention rates,” said Ashok Goel, a Georgia Tech professor who teaches a class entitled Knowledge-Based Artificial Intelligence. “One of the main reasons many students drop out is because they don’t receive enough teaching support. We created Jill as a way to provide faster answers and feedback.”

The class is a core requirement of Georgia Tech’s online master’s program in computer science, and it tends to draw a lot of questions from students. It's offered every semester, and each time, the 300 or so students enrolled post roughly 10,000 messages in the course's online forums, Goel estimates.

That volume has often overwhelmed Goel and his eight teaching assistants, so this time, he added a ninth: Jill.

Jill is a virtual TA, and -- as her name suggests -- she's implemented on IBM’s Watson platform. Goel and his team of graduate students started to build Jill last year, including training her on the roughly 40,000 questions that had been asked in the KBAI class since it was first offered in fall 2014.

“One of the secrets of online classes is that the number of questions increases if you have more students, but the number of different questions doesn’t really go up,” Goel said. “Students tend to ask the same questions over and over again.”

Jill started work as a TA in January, but she wasn’t very good for the first few weeks, often giving odd and irrelevant answers. Her responses were posted in a forum that wasn’t visible to students.

“Initially her answers weren't good enough because she would get stuck on keywords,” said Lalith Polepeddi, one of the graduate students who co-developed the virtual TA.

For example, on one occasion, a student asked about organizing a meet-up to go over video lessons with others, and Jill gave an answer referencing a textbook that could supplement the video lessons: same keywords, but different context.

Further tweaking ensued, and today Jill can answer questions with 97 percent certainty. Initially, the human TAs would upload her successful responses to the students, but by the end of March, Jill didn’t need any assistance: She wrote to the class directly if she was 97 percent positive her answer was correct.

The big reveal didn't happen until late April, when Goel informed his AI students that they had actually been interacting with a bot all semester. One reportedly said her mind was "blown." Since then some students have organized an alumni forum to learn about new developments with Jill after the class ends; another group has launched an open-source project to replicate her.

Next semester, Jill will return with a new name and a new goal: to answer 40 percent of all class questions by the end of the year.