Just when I test and prepare my below comment, shortiiik has posted a similar solution....anyway I am posting here if it helps in some way.....
Yes. UrlReferer is correct first time when page loads, then any action in the page inside web part changes UrlReferer to itself...
I think you need to save the original referer value when page inside the web part loads first time in Session variable and use it later...
protected void Page_Load(object sender, EventArgs e)
{
if (Session["OriginalUrlReferer"] == null) // or use IsPostBack
Session["OriginalUrlReferer"] = System.Web.HttpContext.Current.Request.UrlReferrer;
}
// to use it later...
protected void Button2_Click(object sender, EventArgs e)
{
Uri originalReferer = (Uri)Session["OriginalUrlReferer"];
}