This is something I discussed with some fellow AI enthusiasts 2 years ago. We tried to come up with some "reward function" that would encourage very general behavior and reward more generally intelligent AIs.
The idea I had was to reward collecting blocks of different types, with exponentially decreasing reward. So finding a block it's never found before is 100 points, the second it finds is 50 points, and so on. It encourages exploration and touching on as many mechanics of the game as possible.
The important part is that the AI learns on its own though, not programmed to do specific tasks by a human. However it should perhaps be given recipes and some other meta information, because discovering those by chance is highly improbable.
The idea I had was to reward collecting blocks of different types, with exponentially decreasing reward. So finding a block it's never found before is 100 points, the second it finds is 50 points, and so on. It encourages exploration and touching on as many mechanics of the game as possible.
The important part is that the AI learns on its own though, not programmed to do specific tasks by a human. However it should perhaps be given recipes and some other meta information, because discovering those by chance is highly improbable.