How to use custom master page that in master gallery in application page.

When you add a application page, this page will be exists in _layout folder.
Therefore, you can't use master page in master page gallery.

If you want to use master page gallery's master page,
you should define your master page in OnPreInit event.

protected override void OnPreInit(EventArgs e)
{
base.OnPreInit(e);

SPWeb Web = SPContext.Current.Web;
string strUrl =
Web.ServerRelativeUrl + "/_catalogs/masterpage/default.master";

this.MasterPageFile = strUrl;
}


Please refence MSDN:http://msdn.microsoft.com/en-us/library/cc297200.aspx