Using GetPixel to identify objects inside a game window

Hello friends,

This Tutorial is meant to be beginner friendly, provide information while not being dry as a bone :)

You are probably familiar with the concept of reading pixel color at a position on the screen. This tutorial goes one step further and looks for an "object" of unknown location which can not be identified by a single pixel color.

I also uploaded a video yesterday which includes this example:
[Only registered and activated users can see links. Click Here To Register...]
feedback would be very much appreciated (auch gerne auf Deutsch)

Lets get the basics out of the way:
The following funciton provides pixel color at position x and y.

Code:

COLORREF GetPixel(HDC hdc,int x,int y);

As a third parameter a handle to the device context is needed (hdc).
Since we want to look for something inside a game window we first get a handle to the window and then the desired device context like so:

Code:

HWND hWND = FindWindow(NULL, L"Zuma Deluxe 1.1.0.0");
HDC hDC = GetDC(hWND); //handle to the device context
//Do stuff with hDC
ReleaseDC(hWND,hDC); //make sure to release after use!

Coordinates on the Screen are given like so: Top left is the origin. Going right increases x and going down increases y. Our origin is top left of the game window. We still need the left and bottom border. These we can get with:

Code:

RECT rect = {};
GetClientRect(hWND, &rect);
int right_border = rect.right;
int bottom_border = rect.bottom;

Those are all the basics required.

Now to a practical example:
As my target I picked the game Zuma. What I wanna know is the location of the frog. Starring my enemy right into the eye I realized that those will probably be decent pixels to search for. Turns out that only within those dead eyes there are 2 pixels next to each other having a specific color code:

[Only registered and activated users can see links. Click Here To Register...]

After having found the first eye we already kinda know the frogs position but since the frog can turn 360� I wanted to get a good estimate of the center of the frog. So the idea is to look for the second eye as well then add those two positions up and divide by two to get the center position between those two eyes. The search area for the second eye is restricted:
Since we started top left going down and then right we only need to search to the right of the first eye as well as not too close and not too far away:

[Only registered and activated users can see links. Click Here To Register...]

Equipped with the basics as well as plan for action all that is left to do is write it all down:

Code:

HWND hWND = FindWindow(NULL, L"Zuma Deluxe 1.1.0.0");
std::cout << "looking for frog start" << std::endl;
HDC hDC = GetDC(hWND);
COLORREF color;
COLORREF characteristic_color = 15398653; // Color at the Shiny Spot of the eye
int first_eye_x = 0;
int first_eye_y = 0;
RECT rect = {};
GetClientRect(hWND, &rect); // Stores info about the game window size
for (int i = 0; i < rect.right; i++) { // loop through x coordinates starting at 0
   for (int j = 0; j < rect.bottom; j++) { // loop through y coordinates starting at 0
      color = GetPixel(hDC, i, j);
      if (color == characteristic_color) { // ist this an eye?!
         color = GetPixel(hDC, i, j + 1);
         if (color == characteristic_color) { // same color in the next pixel!! we got an eye
            first_eye_x = i;
            first_eye_y = j;
            std::cout << "first eye: x: " << first_eye_x << "   y:" << first_eye_y << std::endl;
            for (int k = first_eye_x - 35; k < first_eye_x + 35; k++) { // search around first eye for second eye
               for (int l = first_eye_y - 35; l < first_eye_y + 35; l++) {
                  color = GetPixel(hDC, k, l);
                  if (abs(k - first_eye_x) > 10 && abs(l - first_eye_y) > 10 && color == characteristic_color) { // found the second eye!!
                     std::cout << "second eye: x: " << k << "   y:" << l << std::endl;
                     std::cout << "frog center at: x: " << (abs(first_eye_x + k) / 2) << " | y:" << (abs(first_eye_y + l) / 2) << std::endl;
                     return 0;
                  }
               }
            }
         }
      }
   }
}
ReleaseDC(hWND, hDC);

Warning:
This does work BUT I want to give a bit of a warning: GetPixel is a very slow function. So if you need information fast and/or your search area is big this method is probably not a good choice!

Quote:

Originally Posted by Black_and_White

Thanks a lot for the hint. I just had a brief look into at and I must agree It would surely be a safer bet.
Seems like a very good approach for my goal (bot for Zuma) not sure about the required runtime yet. I was planing on looking at loading a "snapshot" of the window into a bitmap next since as far as I know searching through bitmaps in memory is very fast

Yes, loading everything into memory first would probably save a lot of time. The advantage of template matching would be that it's more robust. If the target looks a little different it will still be detected; precise pixel values are not that important.

The downside would be that if the target can rotate you would need to match several rotated versions of the same template. That should not be a big problem though since template matching is quite fast.