A bit of research, and it turns out that 1) there appears to be a bus that goes from the Murayama train station which stops at the Hayashizaki Shrine, and 2) It's probably only about a 2 mile walk. This can be handled.
While doing the research I stumbled on this page, which features some comments by someone who was at the enbu that was done for the All Japan Iaido Federation's 50 year anniversary. It's a little weird looking at the photo and thinking "I was there..."
Another picture from the anniversary festivities
And (because I really am Research Girl), here is a map of the bus routes in Murayama, off of the Murayama City homepage.