zq99 / pgn2data

A library that converts a chess pgn file into a tabulated CSV data set.

Hi there 👋

pgn2data's Issues

Incorrect FENs

Hi zq99, really happy to have found your work. I am having some issues with the following: Imagine this link https://www.chess.com/game/live/76681276597?username=hikaru

which has the following PGN:

[Event "Live Chess"]
[Site "Chess.com"]
[Date "2023.05.01"]
[Round "?"]
[White "Hikaru"]
[Black "Blackmind96"]
[Result "1-0"]
[ECO "B20"]
[WhiteElo "3158"]
[BlackElo "2878"]
[TimeControl "180"]
[EndTime "5:02:13 PDT"]
[Termination "Hikaru won by resignation"]

1. e4 e6 2. d3 d5 3. Nd2 Nf6 4. Ngf3 c5 5. g3 Nc6 6. Bg2 g6 7. O-O Bg7 8. Re1
O-O 9. Qe2 a5 10. Nf1 b6 11. e5 Nd7 12. Bf4 a4 13. h4 a3 14. b3 b5 15. c3 Ba6
16. Rad1 d4 17. c4 Rb8 18. h5 bxc4 19. dxc4 Nb6 20. h6 Bh8 21. N1h2 Nd5 22. Bg5
Nce7 23. Bd2 Nb4 24. Bxb4 Rxb4 25. Ng4 Nf5 26. Nd2 d3 27. Qf3 Bb7 28. Ne4 Bxe4
29. Rxe4 Qg5 30. Qf4 Qxf4 31. Rxf4 Rd8 32. Be4 d2 33. Rf3 Rbb8 34. Rd3 Rxd3 35.
Bxd3 Nd4 36. Kf1 Nc6 37. f4 Nb4 38. Rxd2 Rd8 39. Ke2 Kf8 40. Bb1 Rxd2+ 41. Kxd2
Ke7 42. Nf2 f5 43. exf6+ Bxf6 44. Ne4 Bd4 45. Ng5 Bf2 46. g4 Bd4 47. Nxh7 Bf6
48. Nxf6 Kxf6 49. Ke3 Kf7 50. Ke4 Nc6 51. f5 exf5+ 52. gxf5 g5 53. h7 Kg7 54.
f6+ 1-0

after I get the two cvs with the details and moves I get the column "fen" and is like this:

1. rnbqkbnr/pppppppp/8/8/4P3/8/PPPP1PPP/RNBQKBNR
2. rnbqkbnr/pppp1ppp/4p3/8/4P3/8/PPPP1PPP/RNBQKBNR
3. ...
4. ...
...
Last. 8/6kP/2n2P2/2p3p1/2P1K3/pP6/P7/1B6

Now, this last one is impossible to analyze on stockfish python. It also fails if I put the fen into lichess.org directly.

The correct FEN that you can obtain from the original link says it should be: 8/6kP/2n2P2/2p3p1/2P1K3/pP6/P7/1B6 b - - 0 54

So, how to correct this? I can not do this manually, I am planning on analyzing 30k+ games and of course 100k+ FENs

pgn files where the moves are missing and replaced with a dash, causes the process to stop

If a game has moves that are missing and the move is just a '-', the process crashes with the error:

ValueError: '00' is not in list

This zip file contains an example of a pgn file with missing moves:
missing_moves.zip

Memory leak

I am currently interested in using your librairy to parse png file from chess game.
I am using the lichess database to get my data.
Unfortunatly, when I try to parse it using png2data my RAM constently increase!
I think that the process of reading/writing are maybe not optimized, I think that maybe the data from the read file is still saved.
So for my case, I can't read the total content of a 5GB pgn file.

Unicode Error

Has anyone seen this error before? The file has hundreds of thousands of games but I'm getting a Unicode error when running pgn2data. What I've tired so far is the following. Before I manually look at the pgn file with Scid, any other ideas what could be causing this?

iconv-Linux tool to change the encoding but it fails.

pgn-extract-A pgn command line tool to clean pgn files. Still getting the Unicode error.

I thought about creating a python script in attempt to change the encoding but the solutions I researched were working with the read.csv tool in pandas so I thought that'd be incorrect

OS is Debian 11 Bullseye

Example of error. Had multiple position #'s.

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x94 in position 5478: invalid start byte

Example of iconv error. Sequence position has varied.

iconv: illegal input sequence at position 2635110

Example of the code I was using.

from converter.pgn_data import PGNData

pgn_data = PGNData("multiplegames.pgn")
pgn_data.export()

Only export game_info

I am reading in quite large files and would only require the game_info (not the moves). It would be great if there was an option in the export method to only read the game_info and not the moves.

This would save a lot of disk-space for larger files and possible also speed up the routine.

Clock data stripped

Using this converter on lichess PGN files strips all of the clock data from the PGN lines

work with this pgn structure?

I thought I'd give this a try with a pgn file I have, but I keep getting this error. Any help / pointers appreciated.

Traceback (most recent call last):
  File "chess_export.py", line 6, in <module>
    result = pgn_data.export()
  File "/Users/james/.pyenv/versions/3.7.8/lib/python3.7/site-packages/converter/pgn_data.py", line 59, in export
    result = self.__process_pgn_list(pgn_list, file)
  File "/Users/james/.pyenv/versions/3.7.8/lib/python3.7/site-packages/converter/pgn_data.py", line 91, in __process_pgn_list
    process.parse_file(add_headers)
  File "/Users/james/.pyenv/versions/3.7.8/lib/python3.7/site-packages/converter/process.py", line 96, in parse_file
    game_writer.writerow(self.__get_game_row_data(game, game_id, order, self.pgn_file))
  File "/Users/james/.pyenv/versions/3.7.8/lib/python3.7/site-packages/converter/process.py", line 164, in __get_game_row_data
    game.headers["BlackElo"] if winner == game.headers["Black"] else "")
  File "/Users/james/.pyenv/versions/3.7.8/lib/python3.7/site-packages/chess/pgn.py", line 947, in __getitem__
    return self._others[key]
KeyError: 'BlackElo'

The pgn input file is this

[Date "2022.6.16"]
[Result "0-1"]
[White "Dr. Wolf - Intermediate"]
[Black "James"]

1. c4 1. d5 2. d4 2. e6 3. Nf3 3. c5 4. e3 4. Nc6 5. Nc3 5. Bd6 6. dxc5 6. Bxc5 7. g3 7. dxc4 8. Qa4 8. f6 9. Qb5 9. b6 10. Qxc4 10. Nge7 11. Ne4 11. Qd5 12. Qxd5 12. Nxd5 13. a3 13. f5 14. Bb5 14. Bd7 15. Bxc6 15. Bxc6 16. Ne5 16. Nxe3 17. Nxc6 17. Nc2+ 18. Ke2 18. Nxa1 19. Ng5 19. Nb3 20. Rd1 20. O-O 21. Be3 21. Rfe8 22. Bxc5 22. bxc5 23. Rd7 23. Nc1+ 24. Kd2 24. Nb3+ 25. Kc2 25. Nd4+ 26. Nxd4 26. cxd4 27. Kd3 27. e5 28. b3 28. Rab8 29. Kc4 29. Rec8+ 30. Kd3 30. Rxb3+ 31. Kd2 31. a5 32. Ne6 32. h6 33. Rxg7+ 33. Kh8 34. Rg6 34. Kh7 35. Rg7+ 35. Kh8 36. Rc7 36. Rc3 37. Rxc8+ 37. Rxc8 38. f4 38. e4 39. g4 39. Re8 40. Nc7 40. Re7 41. Nd5 41. e3+ 42. Kd3 42. Re4 43. Nxe3 43. dxe3 44. Ke2 44. fxg4 45. a4 45. Rxa4 46. Kxe3 46. h5 47. f5 47. Ra3+ 48. Kd2 48. Rf3 49. Ke2 49. Rxf5 50. Kd2 50. Rf2+ 51. Kd3 51. Rxh2 52. Ke3 52. a4 53. Kd4 53. g3 54. Kc4 54. a3 55. Kb3 55. a2 56. Kc4 56. a1=Q 57. Kd5 57. Qa2+ 58. Ke4 58. Qc2+ 59. Kf3 59. Qd3+ 60. Kf4 60. Rf2+ 61. Ke5 61. Rf5+ 62. Ke6 62. Qd5+ 63. Ke7 63. Rf7+ 64. Ke8 64. Qd7 0-1

Illegal moves exit execution

When a game within a pgn has an illegal move or a character the codec can't encode, it breaks the whole execution. It seems python-chess raised the error. I handled this issue while working with chess.pgn using:

try:
game = chess.pgn.read_game(pgn_file)
except:
continue

Btw, so much of a great work, you saved me a ton of work:)