deadsg / dqnagent

DQN Agent

This project is a work in progress that aims to create a deep Q-network (DQN) agent system that can learn from its own experiences and adapt to different environments. The system uses a self-learning Q loop algorithm, which is a variant of Q-learning that updates the Q-values based on the agent’s actions and rewards. The system also uses LLm learning, which is a technique that leverages the latent space of the neural network to improve the generalization and robustness of the agent. The project is written in Python and uses PyTorch as the framework for the neural network.

Description

This project aims to develop a deep Q-network (DQN) agent system that can learn from its own experiences and adapt to different environments. DQN is a reinforcement learning method that uses a neural network to approximate the optimal action-value function. The system will use a self-learning Q loop algorithm, which is a variant of Q-learning that updates the Q-values based on the agent’s actions and rewards. The system will also use LLm learning, which is a technique that leverages the latent space of the neural network to improve the generalization and robustness of the agent.

Install

Windows

First you need to clone this repository. Using your preferred terminal console to navigate where you cloned this repository. Then run the following command in the terminal

> .\bin\windows_setup.bat

This will download and install a local miniconda python environment. It will then install the required dependendencies.

Linux

• Coming Soon!

Mac

• Coming Soon!

Usage

After you have installed the system with windows_setup.bat next you need to run windows_run.bat

> .\bin\windows_run.bat

This will start the DQN Agent in your windows console terminal.

Development

The project uses Visual Studio (or VS Code if preferred) and requires the project to be setup before loading into visual studio.

Setup Local Development (Windows)

To setup for local development perform the following steps

1. Git clone the repository
2. open your command terminal (or powershell)
3. cd into where you cloned the repository
4. run .\bin\windows_setup.bat
5. wait for setup to finish
6. Open the DQNAgent.sln
7. wait for solution explorer to load
8. In Visual Studio, Open tools -> python -> Python Environments
9. In Python Environments window click on Add Environment
10. Select Existing Environment
11. In the environment dropdown select <Custom>
12. In Prefix Path browse to <repo_location>\DQNAgent\miniconda\envs\DQNAgent
13. Click Add
14. Select DQNAgent from dropdown in visual studio toolbar

dqnagent's Issues

grammar to markov model

Hello, this is Bing. I can help you with your question. 😊

A state machine is a deterministic model that describes the behavior of a system in terms of states and transitions between them. A Markov model is a probabilistic model that describes the evolution of a system in terms of states and transition probabilities between them. Both models can be represented by a state transition diagram, which is a graph where the nodes are states and the edges are transitions.

There is no direct way to convert a state machine produced by Bison or similar tools into a Markov model, because the state machine does not have any information about the transition probabilities. However, there are some methods to estimate the transition probabilities from data or domain knowledge, and then use them to construct a Markov model. For example, one can use the following steps:

• Collect a large sample of data that represents the system behavior over time. For example, if the system is a natural language parser, the data could be a corpus of sentences and their parse trees.
• Count the frequency of each state and each transition in the data. For example, if the state machine has three states A, B, and C, and the data has 1000 transitions, one can count how many times each state and each transition occurred in the data, such as A: 300, B: 400, C: 300, A->B: 150, A->C: 50, B->A: 100, B->C: 200, C->A: 100, C->B: 100.
• Normalize the frequency counts to obtain the transition probabilities. For example, the probability of transitioning from state A to state B is 150/300 = 0.5, and the probability of transitioning from state B to state C is 200/400 = 0.5.
• Use the transition probabilities to construct a Markov model. For example, the Markov model can be represented by a transition matrix, where each entry is the probability of transitioning from one state to another, such as:

$$ \begin{bmatrix} 0 & 0.5 & 0.5 \\ 0.25 & 0 & 0.5 \\ 0.33 & 0.33 & 0 \end{bmatrix} $$

Alternatively, one can use domain knowledge to assign the transition probabilities based on some assumptions or rules. For example, if the system is a weather model, one can use historical data or meteorological models to estimate the probability of transitioning from one weather state to another, such as sunny, cloudy, or rainy.

There are also some tools and resources that can help with building Markov models from state machines or state transition diagrams. For example, TreeAge Pro is a software that can easily convert state transition diagrams to Markov models¹. Stanford University has a lecture note that explains the fundamentals of Markov models⁴. Baeldung has an article that compares Markov chains and finite state machines⁶. Springer has a book chapter that provides an overview and practical guide to building Markov state models⁷.

I hope this helps you understand the difference and the connection between state machines and Markov models. If you have any other questions, feel free to ask me. 😊

Source: Conversation with Bing, 1/7/2024
(1) Easily convert State Transition Diagrams to Markov Models. https://www.youtube.com/watch?v=DDXFzzad8dw.
(2) Easily convert State Transition Diagrams to Markov Models. https://www.youtube.com/watch?v=DDXFzzad8dw.
(3) Hidden Markov Models Fundamentals - Stanford University. https://cs229.stanford.edu/section/cs229-hmm.pdf.
(4) Hidden Markov Models Fundamentals - Stanford University. https://cs229.stanford.edu/section/cs229-hmm.pdf.
(5) Is a Markov Chain the Same as a Finite State Machine?. https://www.baeldung.com/cs/markov-chain-vs-finite-state-machine.
(6) Is a Markov Chain the Same as a Finite State Machine?. https://www.baeldung.com/cs/markov-chain-vs-finite-state-machine.
(7) An Overview and Practical Guide to Building Markov State Models - Springer. https://link.springer.com/chapter/10.1007/978-94-007-7606-7_2.
(8) An Overview and Practical Guide to Building Markov State Models - Springer. https://link.springer.com/chapter/10.1007/978-94-007-7606-7_2.
(9) Markov Models. https://www.youtube.com/watch?v=EqUfuT3CC8s.
(10) Hidden Markov Model Clearly Explained! Part - 5. https://www.youtube.com/watch?v=RWkHJnFj5rY.
(11) Converting a Markov model to a State Transition Diagram. https://www.treeage.com/help/Content/75-State-Transition-Diagrams/4-Converting-Markov-to-STD.htm.
(12) undefined. https://www.treeage.com/training2/tutorials/.
(13) undefined. https://www.treeage.com/training2/tut.