Image search a window in code

Post here if you need help using Actiona
Post Reply
stifler
Posts: 11
Joined: 29 Oct 2019, 08:51

Image search a window in code

Post by stifler » 01 Nov 2019, 07:57

Hi, back again!

I am trying to implement an image search function in code:

Code: Select all

function Image_Search(ImageWindow, ImageToFind, confidence, pyramids, expansion, maxmatch, mmethod) {
		confidence = typeof confidence !== 'undefined' ? confidence : 95;
		pyramids   = typeof pyramids   !== 'undefined' ? pyramids   : 2;
		expansion  = typeof expansion  !== 'undefined' ? expansion  : 15;
		maxmatch   = typeof maxmatch   !== 'undefined' ? maxmatch   : 1;
		mmethod    = typeof mmethod    !== 'undefined' ? mmethod    : 'CorrelationCoefficient';

		var result = Image.takeScreenshot(ImageWindow).findSubImages(ImageToFind, {
				confidenceMinimum: confidence,
				downPyramidCount: pyramids,
				searchExpansion: expansion,
				maximumMatches: maxmatch,
				method: mmethod,
			});
	return(result);
}
I then try to pass commands to function like so:

Code: Select all

var SomeImage = new Image("C:/Program Files/Actiona/code/picture.png");

var AppWindow = Window.find({
	title: 'MyApplication*',
	titleMode: Window.Wildcard
});

var searchResult = Image_Search(AppWindow,SomeImage);
Debugger claims the problem is in var result = Image.takeScreenshot(ImageWindow).findSubImage(ImageToFind, { line (inside the function Image_Search).
  • Fri Nov 1 16:54:03 2019 Uncaught exception at <anonymous script, id=1667593726064>:271: InvalidWindowError: Invalid window

It would seem I do not understand passing the window details to the function. I see that the result in AppWindow is an array. Is it necessary to take out the title from that array and pass it as a string?
Last edited by stifler on 03 Nov 2019, 01:21, edited 1 time in total.

francois
Posts: 331
Joined: 18 Oct 2010, 10:33
Location: France

Re: Image search a window in code

Post by francois » 01 Nov 2019, 12:14

instead of :

Code: Select all

var searchResult = Image_Search(AppWindow,SomeImage);

Code: Select all

var searchResult = Image_Search(AppWindow[0],SomeImage);
It may be useful to test if the page has been found with something like :

Code: Select all

var w  = Window.find({
title: "*a.htm*",
titleMode: Window.Wildcard
});
switch (true) {
case w.length == 1 :
  var result = Image.takeScreenshot(w[0]);
  Console.print(result);
  break;
case w.length > 1 :  
  Console.print("plusieurs pages correspondent");
  Execution.stop();
  break;
default :
   Console.print(" page non trouvée");
  Execution.stop();

}

stifler
Posts: 11
Joined: 29 Oct 2019, 08:51

Re: Image search a window in code

Post by stifler » 02 Nov 2019, 07:15

Hi Francois,

Code: Select all

var searchResult = Image_Search(AppWindow[0],SomeImage);
The has certainly fixed the debugger error, however I get no result from the search. I confirmed that the correct window is identified and found.

If I do the whole screenshot, it will identify the image it is looking for:

Code: Select all

var r = Image.takeScreenshot().findSubImage(ImageToFind);
The moment I reduce the screenshot to a specific Window, it never finds the image (the error is gone, just no detection).

Code: Select all

var r = Image.takeScreenshot(ImageWindow[0]).findSubImage(ImageToFind);

Console.print(ImageWindow[0]);
Output:
Window  {title: "Heimdall - Google Chrome", className: "Chrome_WidgetWin_1"}
(Confirming the correct window and object being passed to the takeScreenshot function)
So I must still be missing something. The whole screenshot works (which confirms function settings (like pyramid etc) and that the image to find are all correctly configured. It is only when the search is confined by the window coordinates/size (I assume that is what is happening when I tell it a window name).

stifler
Posts: 11
Joined: 29 Oct 2019, 08:51

Re: Image search a window in code

Post by stifler » 03 Nov 2019, 03:07

Further to what I said, I did some additional testing :

Code: Select all

	var s = Image.takeScreenshot(ImageWindow[0]);
	var r = s.findSubImages(ImageToFind, {
		confidenceMinimum: confidence,
		downPyramidCount: pyramids,
		searchExpansion: expand,
		maximumMatches: maxmatch,
		method: mmethod
		});

Console.print(s);
Console.print(r);
This confirms to me that a screenshot of the window has been taken, the result of Console.print(s) reports an image of the correct window size
- Image {width: 638, height: 1049} .

Console.print(r) - on the other hand, stays as undefined. (So "no image found").

So - why does it find my image when I do an entire screenshot, but not find the same image in the specific window, where it is in fact located?

stifler
Posts: 11
Joined: 29 Oct 2019, 08:51

Re: Image search a window in code

Post by stifler » 03 Nov 2019, 06:03

More research.

Code: Select all

var s = Image.takeScreenshot();
var s_file = new File();
var s_file_path = 'C:/Program Files/Actiona/code/~s.bmp';
s_file.open(s_file_path, File.ReadWrite | File.Truncate);
s_file.write(s.data());
s_file.close();
This creates a screenshot of both my monitors and saves it to a file. (about 12Mb)

Code: Select all

var NP_win = Window.find({
	title: 'Notepad*',
	titleMode: Window.Wildcard

var s = Image.takeScreenshot(NP_win[0]);
var s_file = new File();
var s_file_path = 'C:/Program Files/Actiona/code/~s.bmp';
s_file.open(s_file_path, File.ReadWrite | File.Truncate);
s_file.write(s.data());
s_file.close();
Creates a file (about 1.5Mb) but the file is blank (or "all black").

It seems more is needed in code to successfully capture a 'window'. Does it need to be focused foreground first? (even though it is)

stifler
Posts: 11
Joined: 29 Oct 2019, 08:51

Re: Image search a window in code

Post by stifler » 03 Nov 2019, 06:24

Well finally a working version. Though I am not happy with this - since I take a full screenshot, and then I have to use rect function to manipulate it.

Code: Select all

var ss = Image.takeScreenshot();
var s = ss.copy(ImageWindow[0].rect());  // Why! Why do I need to make a copy!

var s_file = new File();
var s_file_path = 'C:/Program Files/Actiona/code/~s.bmp';
s_file.open(s_file_path, File.ReadWrite | File.Truncate);
s_file.write(s.data());
s_file.close();
So I believe there is a bug in the screenshot code.

Or it is not intended to have a window object passed to it. Which would be weird, since it does pick up the dimensions and as I saw in my previous post, it create a bmp file of the correct dimensions as well. But it was a blank image....

francois
Posts: 331
Joined: 18 Oct 2010, 10:33
Location: France

Re: Image search a window in code

Post by francois » 04 Nov 2019, 08:14

Hi,

It works with Window 10, Firefox 70.0.1, Actiona 3.10.0 :

Code: Select all

var SomeImage = new Image("D:/temp/test.jpg");
var w  = Window.find({
title: "*test.htm*",
titleMode: Window.Wildcard
});


if (w.length==0){
    Console.print("'tab' not found")
    Execution.stop();
};


RC = Image_Search(w[0],SomeImage)
if (RC.length==0){
    Console.print("'image' not found")
    Execution.stop();
};

for (i=0;i<RC.length;i++){
  Console.print(RC[i].position.x)
  Console.print(RC[i].position.y)
  Console.print(RC[i].confidence)
}

function Image_Search(ImageWindow, ImageToFind) {
		var result = Image.takeScreenshot(ImageWindow).findSubImages(ImageToFind,            
 {
              confidenceMinimum: 70,
		downPyramidCount: 1,
		searchExpansion: 15,
		maximumMatches: 10,
               method: "CorrelationCoefficient"
 })
return(result);
}

Code: Select all

var NP_win = Window.find({
	title: '*test.htm*',
	titleMode: Window.Wildcard
})

if ( NP_win.length > 0){
var s = Image.takeScreenshot(NP_win[0]);
var s_file = new File();
var s_file_path = 'D:/temp/test_out.bmp';
s_file.open(s_file_path, File.ReadWrite | File.Truncate);
s_file.write(s.data());
s_file.close();
};

stifler
Posts: 11
Joined: 29 Oct 2019, 08:51

Re: Image search a window in code

Post by stifler » 05 Nov 2019, 01:23

Okay, well I am at a loss.

I have now tried both 3.9.4 and 3.10, and I tried both 32 and 64 bit versions of each of those releases. My PC is Win10x64.

So I can only surmise that at this point either something else I am running is interfering (I cannot imagine what), or perhaps its Video card/driver related? (nVidia GTX570).

As I say, full screen capture works.
Any "windowed capture" (I've tried everything that was running, Chrome, notepad++, a security camera app, putty) and they all produce a blank (black) image when targeted. The dimensions are correct, no pic!

I dont know what to try next. I might run up a vm in my vCenter and try on a linux instance, or usb boot Ubuntu on this box to test.

Out of ideas!

stifler
Posts: 11
Joined: 29 Oct 2019, 08:51

Re: Image search a window in code

Post by stifler » 08 Nov 2019, 08:23

Okay!

I found that if I disable Hardware acceleration in Chrome - everything works as expected.

Can the code designer **please** weigh in on this topic.

How do you successfully capture windows in your GUI, when I cannot without disabling hardware accel in Chrome app ?

eureka
Posts: 87
Joined: 08 Mar 2016, 22:18

Re: Image search a window in code

Post by eureka » 10 Nov 2019, 21:29

I am not the code developer but looking at your issue as a problem it occurs to me that you can apply Actiona to toggle off and then toggle on hardware acceleration setting during your main script. I can expand further on the idea but in theory it seems straight forward (although browser has to be reloaded for fresh settings to take effect). You might be better raising the question in the French forum.

Post Reply