No, it has to be not a dll but exe. You can't associate file extension with a dll.
> How would you advise to handle this?
This new exe takes command-line parameter
/open filename
or
/edit filename
and calls the secondary exe, one of those two that you already have.
Or, instead of calling secondary, exe you might transfer your code for each exe into this new exe as subroutine, then there's only one exe.
Or, you change each of the two existing projects from exe to dll, and your new exe calls those dll's.
So, in each case there's new exe that takes parameter /edit or /open, and then calls either separate exe, or dll, or its own internal subroutine.