Android Question VR and device control

[mjtaryan]

I assume, perhaps incorrectly, that control of the device and its software (both OS and apps) are possible at least to some extent. My assumption is based on an app I've installed that performs some control functions such as starting an app, checking for new messages (both sms and email), performing searches. So, my general question is: What control is possible and how?

To be more specific:
1. Is it possible through code for a VR app to interact in various ways with
a. The Notification panel (open, close and clear it, etc.)?
b. The Home screen(s) and the larger (more complete) Laucher (open/close folders, if any, and launch apps?
c. Control another app (for example open it's menu and/or submenu(s) and select menu items, etc.)?

2. If so, are there code snippets for accomplishing these tasks? Where?

I have some more questions related more directly to the VR engine which I will put into another thread.

Thanks.

 

[Erel]

Which app did you install?

Most of these features are not possible with a regular Android app.

 

[mjtaryan]

Which app did you install?

Most of these features are not possible with a regular Android app.

The app is one of the Siri-like clones called "Eva" and is available in the Play Store.

1. It uses ONLY the default Google VR engine.
2. Has a "Speak" bubble rather than the large mic I mentioned in my other VR thread.
3. It can:
a. Launch apps.
b. Send and read instant messages, sms and email.
c. Interact with a user designated Calendar (Add, remove, read and list events).
d. Add and delete (I think) journal/diary entries.
e. Read/display weather info.
f. Peform searches and display results.
and possibly more (I haven't yet explored all of its capabilities).

From what I have explored I get the impression that it can interact with many, if not most, of the built-in apps included in or with Android. I have not experimented with it interacting with apps by other developers (such as the word processor I use).

One thing I'd like to develop is an app that permits full control of the device and its installed apps for those who are visually impaired (including what is generally referred to as a "screen reader" -- which is available on Windows and Mac systems -- these screen readers can "read" whatever is displaayed on the screen, even telling the user if an image is displayed, giving info about the image if that is provided by the developer. Although there are some reader apps for Android, none are as complete and full featured as those for micros. The desire to develop something like this that also uses VR is the motivation for my questions.

 

[Erel]

Many applications allow you to interact with them. From the above list only reading emails is problematic. I wonder whether you need to configure the mail client or it uses the default mail app.

Android has built-in screen reader. You can activate it in the accessibility settings. It is also possible to create a custom accessibility service. However that is currently not supported in Basic4android. You will need to implement it as a library.

 

[mjtaryan]

Many applications allow you to interact with them. From the above list only reading emails is problematic. I wonder whether you need to configure the mail client or it uses the default mail app.

My exploration of this app indicates that it uses the default mail app. I tried getting it to send a gmail message, but it apparently doesn't have "gmail" in its custom command dictionary. In fact, I get the strong impression that it really interacts only with the default apps that are part of the system.

Your first sentence raises two or three questions - How do I determine which apps allow interaction from another app and what interactions a particular app allows? And, once that is determined, how do I accomplish that interaction using B4a? Additionally, you say only reading email is problematic. Since the app does use only the Google VR engine but has replaced the default microphone with a "Speak" bubble that can be set to either the top of bottom of the app, it somehow must be possible to do that. How can we find out how?

Android has built-in screen reader. You can activate it in the accessibility settings. It is also possible to create a custom accessibility service. However that is currently not supported in Basic4android. You will need to implement it as a library.

I used the screen reader is only an example. I am thinking of something both more extensive and moree general. I know a woman who is the victim of the birth control medication Thalidimide (spelling?). She had no hands or feet. For a long time she used a mouth held plastic pipe to use her computer. Recently she started using Nuance's "Naturally Speaking" (I think). I'm envisioning a system that allows full control of a mobile device usisng VR as well as TTS for output. In addition, I envision being able to control bluetooth enabled external devices by VR. For example, were her power wheelchair bluetooth enabled she could tell it what to do through her mobile device.

You say a library would need to be developed. Can such a library be written in B4a or will it need to be in Java (with which I have essentially no experience)? And, if it can be written in B4a, I would need help in learning how to perform the various tasks (as an example for now, taking a VR created text file and incorporating it into the body of an email or sms message.

Thanks ever so much.

 

[Erel]

How do I determine which apps allow interaction from another app and what interactions a particular app allows?

There is no general way to check it. It really depends on each app.

B4A VR feature is based on the standard VR service. With the currently library you cannot replace the dialog.

Adding support for a feature will require writing a Java library.