gammatauai / leetcode-hard-gym Goto Github PK

I find that the latest reflexion repo is not consistent with this one, like PythonSubmissionFormatter vs PySubmissionFormatter.

Hi,

Firstly, nice work with your project.

I notice that in fetch_dataset, the method of obtaining the dataset for leetcode-hard-gym is as follows:

    hard = [q for q in question_infos.stat_status_pairs
            if q.difficulty.level == 3
            and q.paid_only == False]

    uncontaminated = hard[:41] # First question after 41 is contaminated

...

I assume (but not sure if) elements in hard are ordered from newest to oldest problem, but in any case, this code selects the newest 41 (not 40) problems, and will select different problems to those used in the Reflexion paper as long as Leetcode adds new Hard problems.

I checked my generated leetcode-hard-uncontaminated-python3.jsonl, and I don't have the task paths-in-matrix-whose-sum-is-divisible-by-k, while it is present in the one used in Reflexion.

I think that the method used to obtain the problems should ensure the problems generated are consistent to ensure fair comparison between models using this benchmark.

Hi,

I encountered the following API error, have you encountered this and any suggestions on how to fix it?

leetcode.rest.ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Date': 'Fri, 17 Nov 2023 01:02:00 GMT', 'Content-Type': 'application/json', 'Content-Length': '38', 'Connection': 'keep-alive', 'x-frame-options': 'DENY', 'vary': 'Cookie', 'set-cookie': 'LEETCODE_SESSION=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.e30.KpufdHIo8CeGduwC5DCQoba8bmWCjJ9mUTYQ4npFdlk; Domain=.leetcode.com; expires=Fri, 01 Dec 2023 01:02:00 GMT; HttpOnly; Max-Age=1209600; Path=/; SameSite=Lax; Secure', 'strict-transport-security': 'max-age=31536000; includeSubDomains; preload', 'CF-Cache-Status': 'DYNAMIC', 'Report-To': '{"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=LUFCrzz1caB5AX8GXL6VjegtnkfzT1smWIr%2Fzt3ZS%2FusJeoKOQCit7MEq4T2PaQLWnCOCH0H8OiHdKG0fJ%2FVLSxqCVhdaIfm6aaCIgfjz%2BevgOd7QjdR6%2BRbXjBGyQ%3D%3D"}],"group":"cf-nel","max_age":604800}', 'NEL': '{"success_fraction":0,"report_to":"cf-nel","max_age":604800}', 'X-Content-Type-Options': 'nosniff', 'Server': 'cloudflare', 'CF-RAY': '82740bb30d1c62a5-ORD'})
HTTP response body: {"error": "User is not authenticated"}

Best,
Shenao

Thank you so much for the code and your work of Reflexion is inspiring! But I notice there is probably one bug in your source code.

It is likely that you should change the label for CPP in ProgrammingLanguage class in /leetcode_env/types.py like this.

Original Code

class ProgrammingLanguage(Enum):
    """
    Enum for valid LeetCodeSubmission programming languages
    """

    CPP = "c++"
    JAVA = "java"
    PYTHON = "python"
    PYTHON3 = "python3"
    ...

Expected Code

class ProgrammingLanguage(Enum):
    """
    Enum for valid LeetCodeSubmission programming languages
    """

    CPP = "cpp"
    JAVA = "java"
    PYTHON = "python"
    PYTHON3 = "python3"
    ...

The original version will trigger internal server error (status code=500) without useful error hints.
By changing 'c++' to "cpp", the bug is gone.