C# Bejeweled 3 bot - part 1 - tile recognition

As promised earlier here’s how I do screen scrapping and tile recognition in my C# Bejeweled 3 Bot (for obvious reasons I will just insert relevant code snippets, for full source code you will have to see the repo that I will make available soon).

It all starts with an overlay

As you could see in the video I posted in the previous post I start the bot by moving a blue overlay over the game area (the overlay area is actually a form). As you probably saw on the video the form is draggable and I use its position, width and height property for screen grabbing.

Getting an image under the form is awfully simple:

static class ScreenCap
{
  public static Bitmap Grab(Rectangle rect)
  {
    Bitmap bmp = new Bitmap(rect.Width, rect.Height);
    using (Graphics g = Graphics.FromImage(bmp))
    {
        g.CopyFromScreen(rect.X,
                         rect.Y,
                         0, 0,
                         bmp.Size,
                         CopyPixelOperation.SourceCopy);
        return bmp;
    }
  }
}

Since I have the ‘screenshot’ of my game area now I can now start identifying tiles/gems. Since I know the width of the tile (I hardcoded it in the code) I can start my tile identification.

private void setTilesColors(Bitmap bmp)
{
  int offset = TileWidth / 3;
  for (int row = 0; row < boardRows; row++)
    for(int col = 0; col < boardColumns; col++)
    {
      int x = corner.X + row * TileWidth + offset;
      int y = corner.Y + col * TileWidth + offset;
      Rectangle cloneRect = new Rectangle(x, y, TileWidth - 2 * offset, TileWidth - 2 * offset);
      PixelFormat format = bmp.PixelFormat;
      using (Bitmap tile = bmp.Clone(cloneRect, format))
      {
        TileColors[row, col] = getSingleTileAverageColor(tile);
      }
    }
  SimplifyColors(TileColors);
}

What this roughly does is taking the average RGB over the area marked with the yellow cross:

Thinking about it now I probably should find a pixel that has unique color for all the gems but it’s more fun that way! (And I will need average color functions for my other evil projects). Also the center pixel is almost white for all of the gems, so I don’t advise using it!

The TileColors[,] is a two dimensional array containing colors of all the gems on the board.

The SimplifyColor methods goes through all the TileColors[,] elements and makes sure we use the same colors for similar tiles (RGB(126,58,32) is after all different from RGB(123,58,31)).

private void SimplifyColors(Color[,] colors)
{
  CurrentColorList = new List<Color>();
  int width = colors.GetLength(0);
  int height = colors.GetLength(1);
  int threshold = 15;
  for(int row = 0; row < width; row ++)
    for(int col = 0; col < height; col++)
    {
      Color selectedColor = colors[row, col];
      int similarColorIndex = CurrentColorList.GetSimilar(selectedColor, threshold);
      if (similarColorIndex == -1)
      {
          CurrentColorList.Add(colors[row, col]);
      }
      else
      {
          colors[row, col] = CurrentColorList[similarColorIndex];
      }
    }
}

I also created a SimplifyTiles method that takes all the colors on the board, puts them into a list and then fills another 2D array with color id. You can see both colors and their ids on my ‘debug window’.

That’s mostly all when it comes to screen scrapping. In the next post I will explain how I identify possible moves, and what should I improve in the algorithm.

If you want the knowledge right now then jump straight to the repo.