How to reaname the file in both Solidworks and PDM vault?

I believe you should be using IEdmFile6 RenameEx instead of the outdated Rename. You shouldn’t need to check out the files to perform the rename either. See the following thread for more information on why.