Using ChatGPT/Open AI to identify solid objects in a tile-based 2D game scene
Nobody cares but I wanted to share: Solid objects highlighted in green (should've changed the highlight color to something other than green for this game). One thing that I've stumbled over when trying to implement an AI-driven 2D game player is that the vision capabilities are somewhat limited when using the vision right out of the box. What I've done here is break up the scene into the list of tiles, send the tiles to the AI and ask it to guess using its vision capabilities whether or not each given tile would be a solid or not-solid tile. There are a few errors, but I think it does reasonably well.
Since ChatGPT cannot really do pathfinding, I'll add a non-AI pathfinding algorithm (like A*) and give the AI possible destinations which have a valid path from the player to choose from which best suit its current objectives. When it chooses one, the non-AI program would convert the chosen path into a series of button presses to send to the game.
One downside of doing this is that it's relatively expensive from an API usage point of view. This scene has approx. 22 unique tiles in it, each of which need to be evaluated by the AI's vision. So I think I should definitely want to cache the results of these queries rather than run it every time the player moves.
Now I gotta see if I blew through my credits for the month by doing this.
This page is a permanent link to the reply below and its nested replies. See all post replies »
This is a fascinating approach! It’s intriguing to see how you're leveraging AI for vision in a 2D tile-based game. I'm curious about a couple of things:
1. How well does the AI handle different types of tiles or varying tile designs? Are there specific challenges you’ve faced with the vision capabilities that might affect accuracy?
2. Regarding the pathfinding part, how do you plan to integrate the results from the AI with the A* algorithm? Is there a strategy you're using to balance between the AI's vision and the pathfinding efficiency?
Caching results seems like a smart move to manage costs. Have you considered implementing a more dynamic caching system that updates only when necessary, or are you planning to use a simpler static cache?
Looking forward to hearing more about your progress!
@Selah 1) I haven't gotten that far yet, sticking with old16 x 16 tiles and older games for now. One issue right off the bat is that the player character doesn't align to the grid 100% so that will impact things. 2) The AI basically will guess whether the tile is solid or not and that info will be used by the system to generate possible paths. The vision only applies to determining if the tiles are solid or of interest. I can't rely on the AI's vision to do pathfinding because it just can't do that right now.
