Did you implement the getRowId() correctly? This is needed to identify the row to select by model.
Or do you mean on clicking the column header all rows get selected? Well this is because on clicking at the column header, the default behaviour is to select every cell of that column. And with using the RowSelectionModel of course all rows get selected. If you want to avoid this, you need to disable the selection behaviour on clicking the column header. This can be done by overriding and modifying the DefaultSelectionBindings.configureColumnHeaderMouseClickBindings() method.