There was a "criminal subculture" at a jail at the time a prisoner was killed by two inmates, an inquest has heard.