The stocker system is one of the Automated Material Handling System(AMHS) in manufacturing system. The stocker mainly consists of one or two rack masters (or cranes) moving along a single track to transport cassettes. Because the stocker system is the primary material handling system, the performance of stocker system directly affects the overall production performance.

Fig 1. Dual stocker system (Source : Hwang(2018))

In practice, each jobs have different priorities. Various products, customer demands, delivery date and production environments make the different level of job priorities. Therefore, we should consider not only efficiency(makespan) but the priority. Besides, a fairness of the scheduling should be considered.

Fig 2. Scheduling example with priority

Several mathematical optimization methods are used to solving many scheduling problems. Dynamic programming is an optimization approach that transforms a complex problem into a sequence of simpler problems. However, as the problem becomes more complex, it is difficult(or impossible) to find an optimal solution within an appropriate time(suffering from the curse of dimensionality). So, the (meta)-heuristic algorithm is applied to scheduling problem and has the advantage of giving a (near)-optimal solution in a relatively short time compared to the mathematical optimization methods. Tabu search, simulated annealing and genetic algorithm are representative heuristic algorithm that generate appropriate solutions in reasonable computation time.

Recently, machine learning and deep learning approaches, which use the function approximation with neural network, have been widely studied to solve the scheduling problems. Especially reinforcement learning(RL) that learn a policy based on an action value function. In this work, we will study how to model the dual stocker system scheduling problem as a Markov decision process(MDP) and use the deep reinforcement learning to train a scheduler.(RL agent)

Fig 3. Example of RL architecture in scheduling (Source : Luo(2020))

[1] Minjoon Kim, Jeonghoon Mo, “An efficient prioritized scheduling heuristic algorithm for dual stocker systems”, 한국경영과학회 추계학술대회, 2021