C# Bejeweled 3 bot - part 1 - tile recognition

As promised earlier here’s how I do screen scrapping and tile recognition in my C# Bejeweled 3 Bot (for obvious reasons I will just insert relevant code snippets, for full source code you will have to see the repo that I will make available soon).

It all starts with an overlay

As you could see in the video I posted in the previous post I start the bot by moving a blue overlay over the game area (the overlay area is actually a form). As you probably saw on the video the form is draggable and I use its position, width and height property for screen grabbing.

Getting an image under the form is awfully simple:

static class ScreenCap
  public static Bitmap Grab(Rectangle rect)
    Bitmap bmp = new Bitmap(rect.Width, rect.Height);
    using (Graphics g = Graphics.FromImage(bmp))
                         0, 0,
        return bmp;

Since I have the ‘screenshot’ of my game area now I can now start identifying tiles/gems. Since I know the width of the tile (I hardcoded it in the code) I can start my tile identification.

private void setTilesColors(Bitmap bmp)
  int offset = TileWidth / 3;
  for (int row = 0; row < boardRows; row++)
    for(int col = 0; col < boardColumns; col++)
      int x = corner.X + row * TileWidth + offset;
      int y = corner.Y + col * TileWidth + offset;
      Rectangle cloneRect = new Rectangle(x, y, TileWidth - 2 * offset, TileWidth - 2 * offset);
      PixelFormat format = bmp.PixelFormat;
      using (Bitmap tile = bmp.Clone(cloneRect, format))
        TileColors[row, col] = getSingleTileAverageColor(tile);

What this roughly does is taking the average RGB over the area marked with the yellow cross:

Green Bejeweled Orb

Thinking about it now I probably should find a pixel that has unique color for all the gems but it’s more fun that way! (And I will need average color functions for my other evil projects). Also the center pixel is almost white for all of the gems, so I don’t advise using it!

The TileColors[,] is a two dimensional array containing colors of all the gems on the board.

The SimplifyColor methods goes through all the TileColors[,] elements and makes sure we use the same colors for similar tiles (RGB(126,58,32) is after all different from RGB(123,58,31)).

private void SimplifyColors(Color[,] colors)
  CurrentColorList = new List<Color>();
  int width = colors.GetLength(0);
  int height = colors.GetLength(1);
  int threshold = 15;
  for(int row = 0; row < width; row ++)
    for(int col = 0; col < height; col++)
      Color selectedColor = colors[row, col];
      int similarColorIndex = CurrentColorList.GetSimilar(selectedColor, threshold);
      if (similarColorIndex == -1)
          CurrentColorList.Add(colors[row, col]);
          colors[row, col] = CurrentColorList[similarColorIndex];

I also created a SimplifyTiles method that takes all the colors on the board, puts them into a list and then fills another 2D array with color id. You can see both colors and their ids on my ‘debug window’.

Bejeweled 3 grabbed colors
The sole existence of ID 7 on the board indicates that I should adjust my threshold! (Or start looking for the perfect pixel)

That’s mostly all when it comes to screen scrapping. In the next post I will explain how I identify possible moves, and what should I improve in the algorithm.

If you want the knowledge right now then jump straight to the repo.

Mateusz Sadowski

Mateusz Sadowski
Robotics/Drones/Software developer

Livox Mid-40 LiDAR review

Recently I've been testing the performance of Livox Mid-40 LiDAR. This blog post contains my thoughts on using this sensor, it's performance and available ROS node. Continue reading

YDLIDAR X4 - ROS review

Published on January 09, 2019

1% For the Planet

Published on January 01, 2019