Don't cast when we can use the type system better

Due to my lack of familiarity with Scala's type system I've used casts in a few places. I'm not sure which ones we can avoid but I'd be desirable to use the type system better instead of casting and losing type safety to a certain degree.