//
you're reading...
Android, How-to's

HTTYR: Android Voice Control

This post is part of a series on how to train your robot.

Note that this example currently only works if you are using a Wifi dongle, and that the post also assumes you have some familiarity with writing Android Java apps: it is not a tutorial on writing Android apps.

It is quite easy to write an Android app that uses Google Speech recognition to control your robot.

We will use the RemoteRequestEV3 class to control the robot. This is new in the leJOS 0.9.0 release. It is needed as RemoteEV3 does not work on Android as Android does not implement Java RMI. RemoteRequestEV3 uses Java object stream to implement the remote API.

Your app will need a button to connect to the EV3. This creates the RemoteRequestEV3 object, and other remote objects needed to implement remote control, such as two remote regulated motors to drive the robot.

Your will also need a button to press when you want to speak a command. This will start the Google speech recognition engine and send your speech sample to the Google server which will reply with the best suggestions for the commands that you give. You can control the number of suggestions returned.

To find the best suggestion, you can iterate through them from the top suggestion downwards, and check if they include expected key words such as “forward”, “backward”, “stop” etc. When you find a good candidate, you can attempt to parse it and execute the command. The Google speech recognizer works best with natural text, so it is better to include redundant words, for example to say “drive forwards please”, rather than “forward”. For that reason, we check that the command includes certain keywords and ignore other words.

The Google servers can be a bit slow to respond, so don’t expect instantaneous responses to your commands. It is a good idea to include a button to stop the robot as that will work faster than a voice command.

The current version of this app cheats by allowing network access in the main thread. This is not a good idea, as it can make your app unresponsive. It is better to use async tasks to implement communication with the robot, and a future version of this post will show you how to do that.

The code to start the Google speech recognition activity, and to parse and execute the commands that it returns, is:

    private int REQUEST_CODE = 100;
    private int MAX_RESULTS = 5;
    
    /*
     * Start the listen activity
     */
    private void listen() {
    	Intent listenIntent = new Intent(RecognizerIntent.ACTION_RECOGNIZE_SPEECH);
    	listenIntent.putExtra(RecognizerIntent.EXTRA_CALLING_PACKAGE, getClass().getPackage().getName());
    	listenIntent.putExtra(RecognizerIntent.EXTRA_PROMPT, "Robot command");
    	listenIntent.putExtra(RecognizerIntent.EXTRA_LANGUAGE_MODEL, RecognizerIntent.LANGUAGE_MODEL_FREE_FORM);
    	listenIntent.putExtra(RecognizerIntent.EXTRA_MAX_RESULTS, MAX_RESULTS);

        startActivityForResult(listenIntent, REQUEST_CODE);
    }
    
    /*
     * Get the listen results
     */
    @Override
    protected void onActivityResult(int requestCode, int resultCode, Intent data) {
        if (requestCode == REQUEST_CODE && resultCode == RESULT_OK) {
            ArrayList suggestions = data.getStringArrayListExtra(RecognizerIntent.EXTRA_RESULTS);
            commands.setAdapter(new ArrayAdapter (this, R.layout.command, suggestions));
            
            for(String cmd: suggestions) {
            	if (validCommand(cmd)) {
            		doCmd(cmd);
                	break;
            	}

            }
        }
        super.onActivityResult(requestCode, resultCode, data);
    }

The validCommand method should check that the command is a good candidate for a valid command, and doCmd is executes the command by translating it to Remote EV3 method calls. The suggested commands are also added to an Android ListView so that you can see all the suggestions returned by the speech recognizer. You can add a click listener to the ListView, so that you can also execute commands by clicking on them.

The complete text on the app is:

package  org.lejos.sample.speech;

import java.io.IOException;
import java.util.ArrayList;
import java.util.List; 

import lejos.remote.ev3.RemoteRequestEV3;
import lejos.robotics.RegulatedMotor;

import org.lejos.sample.speech.R;

import android.annotation.TargetApi;
import android.app.Activity;
import android.content.Intent;
import android.content.pm.ResolveInfo;
import android.os.Bundle;
import android.os.StrictMode;
import android.speech.RecognizerIntent;
import android.view.View;
import android.view.View.OnClickListener;
import android.widget.AdapterView;
import android.widget.AdapterView.OnItemClickListener;
import android.widget.ArrayAdapter;
import android.widget.Button;
import android.widget.ListView;
import android.widget.Toast;
import android.widget.TextView;

@TargetApi(10)
public class SpeechControl extends Activity implements OnClickListener {
	private ListView commands;
	private RemoteRequestEV3  ev3; 
	private RegulatedMotor left, right;
	private TextView intro;
	
    @Override
    public void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.main);
        Button speechBtn = (Button) findViewById(R.id.speech_btn);
        Button connectBtn = (Button) findViewById(R.id.connect_btn);
        Button disconnectBtn = (Button) findViewById(R.id.disconnect_btn);
        Button stopBtn = (Button) findViewById(R.id.stop_btn);
        connectBtn.setOnClickListener(this);
        disconnectBtn.setOnClickListener(this);
        stopBtn.setOnClickListener(this);
        commands = (ListView) findViewById(R.id.command_list);
        intro = (TextView) findViewById(R.id.header_text);
        // Check if speech recognition activity is available
        List activities = getPackageManager().queryIntentActivities
        		(new Intent(RecognizerIntent.ACTION_RECOGNIZE_SPEECH), 0);
        if (activities.size() > 0) speechBtn.setOnClickListener(this);
        else Toast.makeText(this, "Speech recognition not supported", Toast.LENGTH_LONG).show();
        
        // Nasty hack to allow network access in main thread
        StrictMode.ThreadPolicy policy = new StrictMode.ThreadPolicy.Builder().permitAll().build(); StrictMode.setThreadPolicy(policy);
        
        // Choose a command from the list
        commands.setOnItemClickListener(new OnItemClickListener() {
        	public void onItemClick(AdapterView parent, View view, int position, long id) {
            	doCmd((String) ((TextView) view).getText());
            }
        });
    }
    
    // When button is clicked, listen for a command
    public void onClick(View v) {
        if (v.getId() == R.id.speech_btn) listen();
        else if (v.getId() == R.id.connect_btn && ev3 == null) {
            try {
    			ev3 = new RemoteRequestEV3("192.168.0.9");
    			left = ev3.createRegulatedMotor("A", 'L');
    			right = ev3.createRegulatedMotor("B", 'L');
    		} catch (IOException e) {
    			Toast.makeText(this, "Could not connect to EV3", Toast.LENGTH_LONG).show();
    		}	
        } else if (v.getId() == R.id.disconnect_btn && ev3 != null) {
        	left.close();
        	right.close();
        	ev3.disConnect();
        	ev3 = null;
        } else if (v.getId() == R.id.stop_btn && ev3 != null) {
        	left.stop(true);
        	right.stop(true);
        }
    }

    private int REQUEST_CODE = 100;
    private int MAX_RESULTS = 5;
    
    /*
     * Start the listen activity
     */
    private void listen() {
    	Intent listenIntent = new Intent(RecognizerIntent.ACTION_RECOGNIZE_SPEECH);
    	listenIntent.putExtra(RecognizerIntent.EXTRA_CALLING_PACKAGE, getClass().getPackage().getName());
    	listenIntent.putExtra(RecognizerIntent.EXTRA_PROMPT, "Robot command");
    	listenIntent.putExtra(RecognizerIntent.EXTRA_LANGUAGE_MODEL, RecognizerIntent.LANGUAGE_MODEL_FREE_FORM);
    	listenIntent.putExtra(RecognizerIntent.EXTRA_MAX_RESULTS, MAX_RESULTS);

        startActivityForResult(listenIntent, REQUEST_CODE);
    }
    
    /*
     * Get the listen results
     */
    @Override
    protected void onActivityResult(int requestCode, int resultCode, Intent data) {
        if (requestCode == REQUEST_CODE && resultCode == RESULT_OK) {
            ArrayList suggestions = data.getStringArrayListExtra(RecognizerIntent.EXTRA_RESULTS);
            commands.setAdapter(new ArrayAdapter (this, R.layout.command, suggestions));
            
            for(String cmd: suggestions) {
            	if (validCommand(cmd)) {
            		doCmd(cmd);
                	break;
            	}

            }
        }
        super.onActivityResult(requestCode, resultCode, data);
    }
    
    // Send a command to the EV3
    private void doCmd(String msg) {
    	if (ev3 != null) {
    		ev3.getAudio().systemSound(1);
    		intro.setText("Battery: " + ev3.getPower().getVoltage());
    		if (msg.contains("forward")) {
    			left.forward();
    			right.forward();
    		} else if (msg.contains("backward")) {
    			left.backward();
    			right.backward();
    		} else if (msg.contains("stop")) {
    			left.stop(true);
    			right.stop(true);
    		} else if ((msg.contains("rotate") || msg.contains("turn")) && msg.contains("left")) {
    			left.backward();
    			right.forward();
    		} else if ((msg.contains("rotate") || msg.contains("turn")) && msg.contains("right")) {
    			left.forward();
    			right.backward();
    		}
    	} else Toast.makeText(this, "Not connected to EV3", Toast.LENGTH_LONG).show();
    }
    
    private boolean validCommand(String cmd) {
    	return (cmd.contains("stop") || cmd.contains("forward") || cmd.contains("backward") || cmd.contains("rotate"));
    }
}
Advertisements

Discussion

3 thoughts on “HTTYR: Android Voice Control

  1. Where do I get 0.9.0?

    Posted by Mat | 2014/05/07, 20:45
    • Its not out yet, but you can use the latest version of ev3classes and EV3Menu from Git, as they are still compatible with the 0.8.1 SD card.

      Posted by Geek Grandad | 2014/05/07, 21:27
  2. I just got it working, thx a lot!

    Posted by Juanbot | 2016/05/11, 02:30

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

About leJOS News

leJOS News keeps you up-to-date with leJOS. It features latest news, explains cool features, shows advanced techniques and highlights amazing projects. Be sure to subscribe to leJOS News and never miss an article again. Best of all, subscription is free!
Follow leJOS News on WordPress.com
%d bloggers like this: