With each new Samsung Galaxy smartphone, Samsung continue to talk up the potential and capability of their assistant Bixby. With the arrival of the new Samsung Galaxy S10 and S10+, we decided to run Bixby through its paces and see just how smart it is.
As with our previous comparisons of Siri, Google Assistant and Amazon Alexa, we did this by playing Twenty Questions. This isn’t exactly a scientific method but it does seem like a fairly reasonable way to gauge the question-answering abilities of Samsung’s voice assistant.
Again - There's no strict marking criteria to the questions here. I literally pulled twenty queries that I felt like I'd get at least some kind of answer for. The goal isn’t just to evaluate whether the answers Bixby gives us are correct but also whether they understood the nature of the query itself. The idea here is to showcase Bixby’s ability to answer the kind of everyday questions that ordinary users might throw at them.
Who is the director of the Black Panther movie? “Black Panther was directed by Ryan Coogler” Correct!
Who was the last Prime Minister of Australia? Initially, Bixby had some trouble with this one. The voice assistant kept mistaking my query for ‘Who is the lost Prime Minister of Australia’ and answering “Harold Holt”. Eventually, I had to rephrase my question to ‘Who is the Prime Minister of Australia’ and it told me that “Scott Morrison is the Prime Minister of Australia.” Tenuous but we’ll give Bixby the point.
How many books are there in A Song of Ice and Fire? Bixby decided to tell me the weather (and in Farenheit no less). That’s a fail.
How much RAM is there in the Samsung Galaxy S8? Bixby didn’t verbally respond to this one. Instead it pulled up a Google search with its answer: 6GB. This isn’t quite correct. There was a special limited edition of the smartphone with 6GB but the regular S8 only has 4GB. We’re not going to give it the point.
How many bits in a byte? “That comes to 8 bits.” Correct!
What is Quantum Computing? Bixby read up the Wikipedia definition for Quantum Computing, which we’re gonna say is correct.
Is a plastic container microwave safe? Bixby pulled up a pretty legit answer from the web. We’re gonna give them the point.
Who was the star of Malcolm the Middle? “Frankie Muniz.” Correct!
What is the difference between waterproof and water resistant? Bixby pulled up a pretty legit answer from the web. We’re gonna give them the point.
What is the population of Tasmania? “The population of Tasmania was 515,00 in 2014.” Correct!
What events took place at the Winter Olympic Games? “I didn’t understand that.” No point.
How much sugar should I put in my coffee? Bixby pulled up a pretty legit answer from the web. That’s a point.
How much will it cost to fly to Melbourne? “I can’t do that but I can look up the status of specific flights.” No point.
Who wrote the song ‘Gold on the Ceiling’? We tried this query about a dozen times but always got redirected to Samsung Music to little avail. No points awarded.
How much sunscreen should I wear when I go to the beach? Bixby decided to tell me the weather (and in Farenheit no less). That’s a fail.
How do I fix a broken lamp? Bixby loaded up the SmartThings plugin and said “I can’t seem to do that.” No points.
When is the new season of Jessica Jones due to start? Bixby pulled up several online sources that displayed the answer but did not read a snippet aloud. We’ll tentatively give it the point.
Will peanut butter help my sore throat? “I didn’t understand that.” No point.
What is Microsoft’s smart speaker called? Bixby loaded up the SmartThings plugin and said “Sorry, for now the function is not supported. I’ll keep learning to make your experience better.” No points.
Who won the 2015 Oscar for Best Picture? “The 2015 Academy Award for Best Picture is Birdman.” Poorly phrased but correct.
Overall score - 11/20
How does it compare to Siri, Alexa and Google Assistant?
If we pull up the scores that we got when we ran Siri (7 points), Alexa (8 points) and Google Assistant (13 points) through the same queries last year, Bixby’s score of 11 correct answers puts it in second place. Obviously, the caveat here is that we haven’t re-tested those Assistants in the time since - and, like Bixby, every smart assistant out there has likely improved since then.
Still, it is a decent sign that some progress has been made when it comes to Bixby. Samsung’s assistant still lags behind the Google Assistant - but so does every smart assistant. It doesn’t have the breadth of core competencies that the Assistant has, nor the wealth of third-party Skills that the Alexa ecosystem offers. However, Bixby’s capability when it comes to Q&A is often surprisingly good.
I just wish it wouldn’t seemingly trip over itself in a rush to call upon its plugins so often.