In CMS 5 R2 the UI and Util files have been moved to the Program Files folder and are included with Virtual Path Providers in web.config. Have a look at the section episerver/virtualPath in your web.config file to find out where they are located.
It's of course possible to add your plug-in related files to the Program Files\EPiServer\CMS\... folder found in web.config, but this approach has some implications.
Another alternative is to add your plug-in dependencies to a custom folder in the site, for instance /UIPlugIns/, and then use the overload GetScriptTag(fullPath, false) when adding the script reference to the editor.To properly secure your dialogs you also have to add a location directive in web.config for your plug-in folder with the same security as the UI folder.
<location path="UIPlugIns"> <system.web <authorization> <allow roles="WebEditors, WebAdmins, Administrators"/> <deny users="*"/> </authorization> </system.web></location>
You may also want to have a look at Modal Dialogs in the EPiServer UI for Non-IE Browsers for a run-down on creating modal dialogs in EPiServer CMS 5.
Thanks for this post, been struggling a few hours now and was on the verge of pulling my hairs out.
I got this working by setting my own folder up like Stefan said and adding it manually.