Topics

FW: [program-l] NVDA add-on development questions

 

Hi,

Any ideas?

Cheers,

Joseph

 

From: program-l-bounce@... <program-l-bounce@...> On Behalf Of adil
Sent: Wednesday, May 13, 2020 12:21 PM
To: program-l@...
Subject: [program-l] NVDA add-on development questions

 

Hi all,

I’ve been trying to get my head around with nvda add-on development.

I have already read the nvda developer guide and add-on development guide. So far, I’m able to write simple add-on like saying messages, navigating object hierarchy etc.

 

I’m trying to get the selected text from the browser window.

I tried this code:

selObj = api.getFocusObject().selection

#copying the selected text to clipboard

selObj.copyToClipboard ()

 

the above code works in notepad, but it doesn’t work for browser.

What I know is so far it has to do something with the type of the object return by api.getFocusObjec t ().

For notepad, it returns  NVDAObjects.Dynamic_EnhancedEditFieldIAccessibleEditWindowNVDAObject object

And for browser window, it returns NVDAObjects.IAccessible.ia2Web.Ia2Web object.

 

 

Could someone explain me how do I get the selected text from the browser window?

How do you read the documentation for any class or object as there is no any in detail NVDA documentation?

I tried using pydoc utility to generate documentation from source code, but I just got loss.

Could you explain me how do I use pydoc to read documentation from NVDA source code?

 

Any tips/tricks are welcome!

Bunch of thanks in advance!

 

Sent from Mail for Windows 10

 

** To leave the list, click on the immediately-following link:- ** [mailto:program-l-request@...?subject=unsubscribe] ** If this link doesn't work then send a message to: ** program-l-request@... ** and in the Subject line type ** unsubscribe ** For other list commands such as vacation mode, click on the ** immediately-following link:- ** [mailto:program-l-request@...?subject=faq] ** or send a message, to ** program-l-request@... with the Subject:- faq

Andy B.
 

Hi,

 

They might want to try getting the focusObject’s treeInterceptor, making a textInfos object from it, then getting the selected text from the textInfos object. I haven’t written code for this use case, but it should lead them in the right direction.

 

 

Sent from Mail for Windows 10

 

From: Joseph Lee
Sent: Wednesday, May 13, 2020 3:43 PM
To: nvda-addons@nvda-addons.groups.io
Subject: [nvda-addons] FW: [program-l] NVDA add-on development questions

 

Hi,

Any ideas?

Cheers,

Joseph

 

From: program-l-bounce@... <program-l-bounce@...> On Behalf Of adil
Sent: Wednesday, May 13, 2020 12:21 PM
To: program-l@...
Subject: [program-l] NVDA add-on development questions

 

Hi all,

I’ve been trying to get my head around with nvda add-on development.

I have already read the nvda developer guide and add-on development guide. So far, I’m able to write simple add-on like saying messages, navigating object hierarchy etc.

 

I’m trying to get the selected text from the browser window.

I tried this code:

selObj = api.getFocusObject().selection

#copying the selected text to clipboard

selObj.copyToClipboard ()

 

the above code works in notepad, but it doesn’t work for browser.

What I know is so far it has to do something with the type of the object return by api.getFocusObjec t ().

For notepad, it returns  NVDAObjects.Dynamic_EnhancedEditFieldIAccessibleEditWindowNVDAObject object

And for browser window, it returns NVDAObjects.IAccessible.ia2Web.Ia2Web object.

 

 

Could someone explain me how do I get the selected text from the browser window?

How do you read the documentation for any class or object as there is no any in detail NVDA documentation?

I tried using pydoc utility to generate documentation from source code, but I just got loss.

Could you explain me how do I use pydoc to read documentation from NVDA source code?

 

Any tips/tricks are welcome!

Bunch of thanks in advance!

 

Sent from Mail for Windows 10

 

** To leave the list, click on the immediately-following link:- ** [mailto:program-l-request@...?subject=unsubscribe] ** If this link doesn't work then send a message to: ** program-l-request@... ** and in the Subject line type ** unsubscribe ** For other list commands such as vacation mode, click on the ** immediately-following link:- ** [mailto:program-l-request@...?subject=faq] ** or send a message, to ** program-l-request@... with the Subject:- faq

 

Javi Domínguez
 

Hi.

This addon does it. check it out.

https://github.com/javidominguez/fenreader

    def getSelection(self):
        obj=api.getFocusObject()
        treeInterceptor=obj.treeInterceptor
        if hasattr(treeInterceptor,'TextInfo') and not treeInterceptor.passThrough:
            obj=treeInterceptor
        try:
            info=obj.makeTextInfo(textInfos.POSITION_SELECTION)
        except :
            info=None
        if not info or info.isCollapsed:
            return(None)
        else:
            return(info.text)

Greetings

Javi Dominguez

 
El 13/05/2020 a las 21:43, Joseph Lee escribió:

Hi,

Any ideas?

Cheers,

Joseph

 

From: program-l-bounce@... <program-l-bounce@...> On Behalf Of adil
Sent: Wednesday, May 13, 2020 12:21 PM
To: program-l@...
Subject: [program-l] NVDA add-on development questions

 

Hi all,

I’ve been trying to get my head around with nvda add-on development.

I have already read the nvda developer guide and add-on development guide. So far, I’m able to write simple add-on like saying messages, navigating object hierarchy etc.

 

I’m trying to get the selected text from the browser window.

I tried this code:

selObj = api.getFocusObject().selection

#copying the selected text to clipboard

selObj.copyToClipboard ()

 

the above code works in notepad, but it doesn’t work for browser.

What I know is so far it has to do something with the type of the object return by api.getFocusObjec t ().

For notepad, it returns  NVDAObjects.Dynamic_EnhancedEditFieldIAccessibleEditWindowNVDAObject object

And for browser window, it returns NVDAObjects.IAccessible.ia2Web.Ia2Web object.

 

 

Could someone explain me how do I get the selected text from the browser window?

How do you read the documentation for any class or object as there is no any in detail NVDA documentation?

I tried using pydoc utility to generate documentation from source code, but I just got loss.

Could you explain me how do I use pydoc to read documentation from NVDA source code?

 

Any tips/tricks are welcome!

Bunch of thanks in advance!

 

Sent from Mail for Windows 10

 

** To leave the list, click on the immediately-following link:- ** [mailto:program-l-request@...?subject=unsubscribe] ** If this link doesn't work then send a message to: ** program-l-request@... ** and in the Subject line type ** unsubscribe ** For other list commands such as vacation mode, click on the ** immediately-following link:- ** [mailto:program-l-request@...?subject=faq] ** or send a message, to ** program-l-request@... with the Subject:- faq


James Scholes
 

As others have noted, lots of add-ons already do this e.g. Instant Translate. But overall, the best suggestion you can give this person is to join this list and then ask again.

Regards,

James Scholes

On 13/05/2020 at 2:43 pm, Joseph Lee wrote:
Hi,
Any ideas?
Cheers,
Joseph
*From:* program-l-bounce@... <program-l-bounce@...> *On Behalf Of *adil
*Sent:* Wednesday, May 13, 2020 12:21 PM
*To:* program-l@...
*Subject:* [program-l] NVDA add-on development questions
Hi all,
I’ve been trying to get my head around with nvda add-on development.
I have already read the nvda developer guide and add-on development guide. So far, I’m able to write simple add-on like saying messages, navigating object hierarchy etc.
I’m trying to get the selected text from the browser window.
I tried this code:
selObj = api.getFocusObject().selection
#copying the selected text to clipboard
selObj.copyToClipboard ()
the above code works in notepad, but it doesn’t work for browser.
What I know is so far it has to do something with the type of the object return by api.getFocusObjec t ().
For notepad, it returns
NVDAObjects.Dynamic_EnhancedEditFieldIAccessibleEditWindowNVDAObject object
And for browser window, it returns NVDAObjects.IAccessible.ia2Web.Ia2Web object.
Could someone explain me how do I get the selected text from the browser window?
How do you read the documentation for any class or object as there is no any in detail NVDA documentation?
I tried using pydoc utility to generate documentation from source code, but I just got loss.
Could you explain me how do I use pydoc to read documentation from NVDA source code?
Any tips/tricks are welcome!
Bunch of thanks in advance!
Sent from Mail <https%20:/go.microsoft.com/fwlink/?LinkId=550986> for Windows 10
** To leave the list, click on the immediately-following link:- ** [mailto:program-l-request@...?subject=unsubscribe] ** If this link doesn't work then send a message to: ** program-l-request@... <mailto:program-l-request@...> ** and in the Subject line type ** unsubscribe ** For other list commands such as vacation mode, click on the ** immediately-following link:- ** [mailto:program-l-request@...?subject=faq] ** or send a message, to ** program-l-request@... <mailto:program-l-request@...> with the Subject:- faq

Adil Shaikh
 

thanks to everyone who answered my query. 
I got that part of the add-on working. now, I am trying to add the selected text to a third party application.  initially, I  was considering to use clipboard to pass the selected text to third party application by just using simple copy and paste. but it doesn't feel quite right to me as it messes up with the clipboard content.  
I've already tried this code:
editBox = api.getFocusObject ()
editBox.value = "some text
it's not working. 

thanks josephly for forwarding my post to this mailing list. 

James Scholes
 

Send a Windows WM_SETTEXT message to the field, assuming that it's a standard Win32 application.

import winUser
editBox = api.getFocusObject()
WM_SETTEXT = 0x000C
winUser.sendMessage(editBox.windowHandle, WM_SETTEXT, 0, "test")

You may also need to send an EM_SETSEL message to move the cursor after the text if required, because I think by default the cursor will remain at the start.

https://docs.microsoft.com/en-us/windows/win32/controls/em-setsel

Regards,

James Scholes

On 15/05/2020 at 1:44 pm, Adil Shaikh wrote:
thanks to everyone who answered my query.
I got that part of the add-on working. now, I am trying to add the selected text to a third party application.  initially, I  was considering to use clipboard to pass the selected text to third party application by just using simple copy and paste. but it doesn't feel quite right to me as it messes up with the clipboard content.
I've already tried this code:
editBox = api.getFocusObject ()
editBox.value = "some text
it's not working.
thanks josephly for forwarding my post to this mailing list.

Adil Shaikh
 

thanks!
it works like a charm.

Javi Domínguez
 

Hello, Adil.

I think that's a typical beginner mistake. You've done:

editBox = api.getFocusObject ()

and you thin that editBox it is the authentic, original object, that of the applicationn but no, editBox is a virtual object created by NVDA with the attributes that it has been able to extract from the application object.

All NVDAObjects are reflections of the original objects, you can't act on the object of the application modifying the NVDA objects.

If you do:

editBox.value = "some text

you make the user see "some text", but the edit box in the screen, the reflected object, the source, I don't know how to say it, doesn't change.

So you have to call the operating system through the winUser module, as James has explained to you.

This is what I think but there are people here who know much more than I do. Correct me if I'm wrong, please.

Greetings

Javi Dominguez


El 15/05/2020 a las 20:44, Adil Shaikh escribió:
thanks to everyone who answered my query. 
I got that part of the add-on working. now, I am trying to add the selected text to a third party application.  initially, I  was considering to use clipboard to pass the selected text to third party application by just using simple copy and paste. but it doesn't feel quite right to me as it messes up with the clipboard content.  
I've already tried this code:
editBox = api.getFocusObject ()
editBox.value = "some text
it's not working. 

thanks josephly for forwarding my post to this mailing list. 


 

Hi,

The following message will contain geeky material, so please hold on:

An NVDA object is NVDA’s way of looking at graphical elements. A given NVDA object has a name, role, value, states, location, among other things. But in order to represent what is seen as “real” by users, it also remembers what kind of object it is: MSAA/IAccessible, UI Automation, JAB, and many others, as evidenced by developer info listing attributes coming from these API’s. As Javi pointed out, changing the attributes of NVDA objects will not change what accessibility API’s say about a particular control – it just tells NVDA objects to act like something else, give users something other than what is actually shown on screen, among other tricks.

Technical: an NVDA object is an abstract representation of a user interface element on screen – visible or invisible, differing roles, across apps, across accessibility API’s, and many other things. A base class named NVDAObjects.NVDAObject is responsible for specifying the blueprints of this abstract class: name, role, states, app module, events, among others. Although the base NVDA object class isn’t marked as an abstract base class (ABC), you don’t really want to instantiate this class (technically, you can but under specific circumstances such as providing gestures), and several descriptors in this base class are marked with “NotImplementedError”.

Because the base NVDA object class is primitive at best, you won’t find them in real life; rather ,you’ll interact with one concrete implementation, which serves as the “real” NVDA object: NVDAObjects.window.Window. Every control you see (or can interact with) on screen is a window (including windows that are invisible). Apart from specific exceptions, every window is represented by a window handle (hwnd). Window handles shield apps (including NVDA) from direct manipulation, and without this shield, it introduces security problems. Window handles are also important for NVDA (and the next component described below) as NVDA can use it to query certain things about a window or a control such as window text and such. Unless you need a way to direct Windows to talk to controls in a controlled fashion, you won’t have to interact with window objects.

Window handle is also needed to construct what’s termed “API class”. An API class is a specification of an NVDA object based on an accessibility API’s notion of an element. Each API class (or for that matter, accessibility API’s) have differing notions as to what the screen element should say to screen readers (technically termed “accessibility clients”) and how apps and assistive technologies communicate. In short, an accessibility API such as MSAA and UI Automation serves as a “bridge” between apps (termed “servers”) and screen readers such as NVDA (clients), and as such, one of their duties is to provide information about a graphical element to NVDA based on what the app reports about a given control (now you know why I and others get frustrated about UIA implementation issues from Microsoft). Again, without a window handle for the element in question, accessibility API’s cannot do its job, and for NVDA, it cannot instantiate API classes.

Another thing an API class must have is a way to talk to the control in question via accessibility API. Each API class includes a reference to the control as seen by accessibility API. For MSAA/IAccessible, it is object.IAccessibleObject; for UIA, it is object.UIAElement. And since API classes are objects representing a control as reported by the app through accessibility API, NVDA receives directions from API’s as to how to present things such as name, role, states, location, value, among other things. Because apps are imperfect (think about what I just said for a second before moving on), information reported by IAccessible/UIA/JAB might be incomplete or incorrect; for this reason, NVDA employs tricks to provide more accurate information to the best of a developer’s knowledge (for a practical example, see Windows 10 App Essentials add-on, and many other examples from community add-ons website). And this last family of classes is what NVDA ships with, and can be used by add-ons for specific purposes (this last point is called an “overlay class”).

There is a special family of NVDA objects that doesn’t take a window handle as a required constructor argument: behaviors. Behavior classes are used to teach NVDA as to how to behave for specific controls such as terminal windows (announce incoming text automatically if given a chance), notification messages and alerts, table navigation, and auto-suggest accessibility (playing a sound when suggestions appear). Because behavior classes are meant to enhance user experience, they don’t require window handles or other fancy footwork to get them to work; rather, they are added to existing API/overlay classes (see UIA console code for practical examples).

Cheers,

Joseph

 

From: nvda-addons@nvda-addons.groups.io <nvda-addons@nvda-addons.groups.io> On Behalf Of Javi Domínguez
Sent: Sunday, May 17, 2020 1:18 PM
To: nvda-addons@nvda-addons.groups.io
Subject: Re: [nvda-addons] FW: [program-l] NVDA add-on development questions

 

Hello, Adil.

I think that's a typical beginner mistake. You've done:

editBox = api.getFocusObject ()

and you thin that editBox it is the authentic, original object, that of the applicationn but no, editBox is a virtual object created by NVDA with the attributes that it has been able to extract from the application object.

All NVDAObjects are reflections of the original objects, you can't act on the object of the application modifying the NVDA objects.

If you do:


editBox.value = "some text

you make the user see "some text", but the edit box in the screen, the reflected object, the source, I don't know how to say it, doesn't change.

So you have to call the operating system through the winUser module, as James has explained to you.


This is what I think but there are people here who know much more than I do. Correct me if I'm wrong, please.

Greetings

Javi Dominguez

El 15/05/2020 a las 20:44, Adil Shaikh escribió:

thanks to everyone who answered my query. 
I got that part of the add-on working. now, I am trying to add the selected text to a third party application.  initially, I  was considering to use clipboard to pass the selected text to third party application by just using simple copy and paste. but it doesn't feel quite right to me as it messes up with the clipboard content.  

I've already tried this code:

editBox = api.getFocusObject ()

editBox.value = "some text

it's not working. 

thanks josephly for forwarding my post to this mailing list. 

 

Adil Shaikh
 

Thanks for the explanation,

It explains me how it all works.

I’m just trying  to summarize your explanation to solidify my understanding.

 

1.       Every window control is represented by NVDAObject

2.       To create NVDAObject, window handle is obtain from the application

3.        This window handle  is passed to the accessibility api (JAB MSAA etc)

4.       The accessibility api expose the window by different methods/properties

5.        The accessibility client receives information and creates object (in this case  NVDAObject.IAccessible etc )

 

Another thing an API class must have is a way to talk to the control in question via accessibility API. Each API class includes a reference to the control as seen by accessibility API. For MSAA/IAccessible, it is object.IAccessibleObject; for UIA, it is object.UIAElement.

I’m bit confused here. Do you mean the accessibility API can provide mechanism  to modify source object (the real application window?

 

Please correct me if I misunderstood something.

 

Best wishes

James Scholes
 

Hi Adil,

The content of Joseph's earlier message is useful for background, if you truly wish to know how the various parts of NVDA, your operating system, etc. come together to provide you with an accessible experience. But in this case, the most important thing to take away from it is that you cannot set the value of an edit field by modifying the NVDA object representation of it.

I did send a separate message to the list, explaining how you can set the text of a standard Windows edit field. Did you try that? If it doesn't work, or if the application is not standard Win32, then you may not be able to achieve what you want from within NVDA. For example, if you're trying to set the content of an edit field on a web page, you definitely should be writing a browser extension instead, not an NVDA add-on.

If you need more info, please tell us what you're trying to achieve, and in which application.

Regards,

James Scholes

On 19/05/2020 at 10:02 am, Adil Shaikh wrote:
Thanks for the explanation,
It explains me how it all works.
I’m just trying to summarize your explanation to solidify my understanding.
1.Every window control is represented by NVDAObject
2.To create NVDAObject, window handle is obtain from the application
3.This window handle is passed to the accessibility api (JAB MSAA etc)
4.The accessibility api expose the window by different methods/properties
5.The accessibility client receives information and creates object (in this case NVDAObject.IAccessible etc )

Another thing an API class must have is a way to talk to the control in question via accessibility API. Each API class includes a reference to the control as seen by accessibility API. For MSAA/IAccessible, it is object.IAccessibleObject; for UIA, it is object.UIAElement.

I’m bit confused here. Do you mean the accessibility API can provide mechanism to modify source object (the real application window?
Please correct me if I misunderstood something.
Best wishes

Adil Shaikh
 

thank you James, 
your solution had worked for me. I've already understood that I can't set the value of edit box using NVDA object. 
I've been trying to launch app(wordweb dictionary) from my script. I haven't found the way. I looked at winUser module, but couldn't find any method/class that can do it. 

best wishes 

James Scholes
 

import os
os.startfile('path to .exe of dictionary app')

Regards,

James Scholes

On 19/05/2020 at 12:19 pm, Adil Shaikh wrote:
thank you James,
your solution had worked for me. I've already understood that I can't set the value of edit box using NVDA object.
I've been trying to launch app(wordweb dictionary) from my script. I haven't found the way. I looked at winUser module, but couldn't find any method/class that can do it.
best wishes

Adil Shaikh
 

thanks James, it works.
👍👍