Welcome to the captivating world of Reinforcement Learning (RL), where we delve into the groundbreaking concept of curiosity-driven exploration introduced in a seminal paper presented at ICML 2017. Today, we’ll guide you through the intricacies of using intrinsic curiosity-based motivation (ICM) to train agents, particularly when external rewards are sparse or absent.

What is Curiosity-driven Exploration?

The idea behind this innovative approach is straightforward yet powerful: when external rewards are scarce, an agent can still be motivated to explore its environment purely out of curiosity. Imagine letting a child wander through a museum, every corner offering something new to learn, even without structured lessons. This exploration enhances their understanding and skills, much like how our RL agents learn from their environments.

How to Install and Use the Code

Setting up the TensorFlow-based implementation is a seamless experience if you follow these structured steps:

1) Installation Steps

• Run the following shell commands in your terminal:

sudo apt-get install -y python-numpy python-dev cmake zlib1g-dev libjpeg-dev xvfb libav-tools xorg-dev python-opengl libboost-all-dev libsdl2-dev swig python3-dev python3-venv make golang libjpeg-turbo8-dev gcc wget unzip git fceux virtualenv tmux

• Clone the repository:

git clone -b master --single-branch https://github.com/pathak22/noreward-rl.git

• Navigate to the project directory and set up a virtual environment:

cd noreward-rl
virtualenv curiosity
source $PWD/curiosity/bin/activate
pip install numpy
pip install -r src/requirements.txt
python curiosity/src/go-vnc-driver/build.py

• Download models:

bash models/download_models.sh

• Set up your customized Doom environment:

cd doomFiles
# Follow commands in doomFiles/README.md

2) Running the Demo

For a hands-on look, run the demo:

• Navigate to the demo source directory:

cd noreward-rl/src
python demo.py --ckpt ../models/doom/doom_ICM
python demo.py --env-id SuperMarioBros-1-1-v0 --ckpt ../models/mario/mario_ICM

3) Training Your Model

Ready to train your model? Follow these steps:

• For Doom, use:

cd noreward-rl/src
python train.py --default --env-id doom

• For Mario, first adjust the constants:

# PREDICTION_BETA = 0.2
# ENTROPY_BETA = 0.0005
python train.py --default --env-id mario --noReward
xvfb-run -s -screen 0 1400x900x24 bash
# only for remote desktops

• Finally, execute the inference:

python inference.py --default --env-id doom --record

Troubleshooting

If you encounter any issues during installation or execution, consider these troubleshooting tips:

• Ensure all dependencies are properly installed, particularly those related to TensorFlow.
• If a command fails, check your syntax and ensure paths are correct.
• Consult the project website or demo video if you need visual guidance on setup.
• For more insights, updates, or to collaborate on AI development projects, stay connected with fxis.ai.

Further Learning Resources

For those interested in delving deeper, here are additional resources:

• Paper
• Project Website
• Demo Video
• Reddit Discussion
• Media Articles

In conclusion, at fxis.ai, we believe that such advancements are crucial for the future of AI, as they enable more comprehensive and effective solutions. Our team is continually exploring new methodologies to push the envelope in artificial intelligence, ensuring that our clients benefit from the latest technological innovations.

Get Started!

With this guide, you’re well-equipped to dive into curiosity-driven exploration in reinforcement learning. Happy coding!