Aiming at the problems of ecological environment left behind by abandoned mines, the abandoned gold-mercury mine in Danzhai, Guizhou Province was selected as the research area. Based on the field systematic sampling, the concentrations of mercury (Hg) and arsenic (As) in the soils and crops were analyzed by atomic fluorescence spectrometry and inductively coupled plasma mass spectrometry. The heavy metal pollution level and ecological health risk of soils and crops were evaluated by single factor standard index, single factor index method, potential ecological hazard index and target hazard quotient. The results show that the average concentrations of Hg and As in dryland soil exceed the screening value of heavy metals in agricultural soils, with the exceeding rates of 75.47% and 67.92% respectively. Dryland soil Hg is seriously polluted, showing extremely heavy pollution level, while soil As is mainly medium to extremely heavy pollution. The exceeding rates of Hg and As in aboveground parts of corn (hereinafter referred to as corn) are 36.36% and 4.55% respectively. Corn Hg is slightly polluted, and corn As is not polluted in general. Compared with the dryland-corn system, the Hg over-standard rate and pollution degree of paddy soil are higher, with the exceeding rate of 89.19%, showing an extreme pollution level. However, the over-standard rate of As in paddy soil is only 22.22%, and the As pollution level is lower than that of dryland soil. The average Hg and As concentrations of aboveground rice (hereinafter referred to as rice) are slightly higher than the food safety standards, and the exceeding rates are 54.55% and 18.18% respectively. The Hg and As pollution levels of rice keep higher than those of corn, and the pollution levels is in the light to medium range. The potential ecological hazard index shows that Hg ecological risk in soils is mainly extremely strong, and As ecological risk is mainly slight. It is difficult for Hg to migrate to crop seeds, so there is no obvious non-carcinogenic risks by eating corn and rice. However, the accumulation of As in rice can result in significant non-carcinogenic and carcinogenic health risks. This study is of great significance to ensure the health of mining residents, and the results can provide scientific basis for the ecological environment impact assessment and ecological restoration in mining areas.