Chessvision.ai eBook Reader (January 2020)

tl;dr I made chess eBook Reader that uses computer vision and deep learning to make chess ebooks interactive. Open an ebook, send it to analysis just the first time, once finished, double-click on any chess diagram in the book and explore it on the analysis board.

App URL: https://ebook.chessvision.ai/

After I proposed the idea and outstanding beta tests in November 2019 - thanks to all people who helped with beta tests - you are awesome, I could not make it without your effort and support! - I finally published the app in January 2020 - launch on Reddit

It is an eBook Reader for chess ebooks specifically. It makes any PDF chess ebook interactive in that sense that once a book is analyzed by the app, you can double-click on any chess diagram in the book and it will show up the analysis board where you can make your moves, turn the engine on/off, see top 5 computer lines which most often contains lines in the book, see related games where the recognized position occurred on the board, and more. Every book is analyzed only once, the next time you open it, you can interact with it immediately.

The app is completely free to try so everyone can check if it is suitable for their studies and preferences, and only then decide if the member account helps with that. The only limitation of a free account is that the diagrams are interactive on only over a dozen of first pages containing diagrams in a book.

If you one does not have any chess ebooks to try, there are plenty of free samples on the internet, e.g. https://www.qualitychess.co.uk/pdf.htm

As of March 2020, the app has just over 300 active users, including titled chess players and chess teachers.

Chess.com Voice Commentary (December 2019)

I teamed up with Chess.com to build an automatic commentary app for live games played on chess.com. It was a hell of a fun experience and I was responsible for all the programming. The app observes the state of a live game and captures events like moves, opening details, being low on time, thinking for a long time, game result, and translates them to corresponding sounds combinations that are played in response to an event. Chess.com team delivered many various sound samples and one fun task was implementing an algorithm that maps a move played in a game to the best sequence of sounds to combine to represent that move.

Download: Chrome extension

GANChessSet (August 2019)

Having collected quite large and a representative dataset containing various chess pieces of different styles and sizes. I started experimenting with generative adversarial networks in the hope to generate interesting results. For training, I picked ~35k black pieces.

Highlights

From 64k new synthesized images, I picked ones I like the most:

Unicorn

Plague Doctor

Recursive Bishop

Knishop

Cobra

Stylish Pawn

Sneaky Knight

Monster

Full generated set

Grid with 64000 images. Open and zoom to see individual ones. Every picture is numbered in its upper left corner

Reddit discussion

I posted the results on Reddit. The post was much discussed and people picked their favorite pieces. Used Neural Networks to generate chess pieces, which are your favorite ones?

The next step is to use the results to create a full chess set that can be used for online play on chess.com or/and lichess.org

chessvision-ai-bot for Reddit (April 2019)

Being an active user on r/chess subreddit, I observed that a significant amount of posts there have images with chess positions, either screenshots from games, books, etc., puzzles, positions from books and others.

Since I already have a full pipeline to recognize 2d chess position from images, I thought that I can write a bot for that subreddit that can observe new posts posted there, analyze images the posts contain and reply with comments containing links to board analysis corresponding to the position on examined images.

Discussion on Machine Learning subreddit: [P] I'm a bot and will serve people analyzing chess positions from images posted on /r/chess

Highlights

Funny Highlights

Engine hints (June 2019)

For the bot to be even more useful, I added the engine moves as hints marked with spoilers directly in the post. To achieve that, I implemented a separated REST API that takes a position and returns Stockfish evaluation. The service behind the API is implemented in Javascript and as far I know this is the only such service available. However, I didn't make it public because of the heavy computation it involves. I'd be happy to give access to anyone who wants to use it in a reasonable way.

Reddit discussion about this feature: u/chessvision-ai-bot provides hints to the positions he analyzes

Currently, in order to provide hints, the bot tries to predict board orientation using simple heuristic rule-based on kings' positions. That works most of the time except some of the endgames when kings march up the board. I'm going to implement a more accurate way to get the correct board orientation using the coordinates on the image if they are present.

chessvision.ai (April 2019)

Initially, I wanted to pick up an image classification task that is both relatively easy to start with, challenging to do very accurately and brings value to people. I thought that recognizing chess positions from images is a perfect such task and started working on it.

Algorithms

The task quickly decomposed into two separated problems

Detecting chessboards in images
Most of the other existing methods for this task use line detection combined with merging detected lines in a grid. There are also a few methods that rely on pattern matching but none of them was production-ready. I didn't find those methods accurate enough for my needs and also wanted to build something less restricted and more general and that's how it all started.
My solution is a combination of pattern detection with techniques related to graph algorithms to get the position of a chessboard. It takes an expected size of the grid as an input parameter so it's generic enough to be used to detect any checkerboard-like grids of a given size. Here are some results of the final method applied to various images: chessboards detection results
I hope I'll find time to publish a paper about it as from my research it's an accurate and efficient algorithm to find chess-grids in images and uses new methods in this field.
Classification of individual cells of the chessboard
At first, it might seem a straightforward task not much different than MNIST, but the use case is crucial here. In order to get the full standard chessboard right, we have to classify all 64 cells correctly. Assuming that some classifier on average gets 0.99 accuracy score then roughly we'd get 0.5 chance to get the full board correct because 0.99⁶⁴ ~ 0.52. So the goal was to build a classifier that has a significantly better accuracy score than 0.99.
I achieved it by first decoupling the task of classifying the content into two separated tasks: piece type classification (6 classes + empty square) and color classification of a piece. After that, I had to collect a lot of data that mimics the expected distribution of future input quite well. I started with an automatic generating of labeled data (several piece themes and several board themes combined). After that, I trained the first version of a network on a fairly simple architecture. I used that network to extend the dataset with real, non-generated data, by gathering unlabeled data, label it using the network and fix mislabeled examples by hand. I followed it up with training the next network with that extended huge dataset and a bit deeper architecture. The final step was to see what kind of specific inputs the network performs poorly and fix that with even more data corresponding to those mistakes.

Browser extensions

When I had solutions to both of these ready, I thought that I can make the most of it by publishing a browser extension that allows recognizing chess positions from any website, video or book opened in a browser. After publishing, people started reposting it and it went viral:

Download links:

formatted by Markdeep 1.093

✒