I think a tweak that would make selecting and deselecting objects more intuitive is if, while in "select mode", single pressing (either with pencil or finger) on a blank part of the page would deselect all. It works this way when you're NOT in "select mode" and seems like logically it could work there as well.