To create a modal dialogue box in visualforce page follow below steps.

• Login to your Salesforce.com developer account.
• Click “Setup” in the upper right corner.
• Under the App Setup section (left menu), expand Develop.
• Click on Apex Classes.
• Click the “New” button to create a new Apex Class.
• Add the following source and click the “Save” button.

  public class TestPopup{
      public boolean DisplayPopup {get; set;}    
      
      public void ClosePopup() {       
          DisplayPopup = false;   
      }
   
      public void ShowPopup() {       
          DisplayPopup = true;   
      }
    }
  

• Click on Pages (below Apex Classes on the left menu).
• Create a new Visualforce page by clicking the “New” button.
• For the purpose of this tutorial I used “TestPopupPage” for my Label & Name.
• Enter the Visualforce Markup code below and click “Save”.

  <apex:page controller="TestPopup">
      <apex:form>
          <apex:commandbutton action="{!ShowPopup}" rerender="TestPopup" value="Show Popup">
          <apex:pageblock>
              This is a sample Test Popup Page.
          </apex:pageblock>
          <apex:outputpanel id="TestPopup">
          <apex:outputpanel layout="block" rendered="{!DisplayPopUp}" styleclass="popupBackground">
              <apex:outputpanel layout="block" rendered="{!DisplayPopUp}" styleclass="customPopup">
                  Hi, Biswajeet 
                  <apex:commandbutton action="{!ClosePopup}" rerender="TestPopup" value="Close Popup">
              </apex:commandbutton></apex:outputpanel>
          </apex:outputpanel>
   
      </apex:outputpanel></apex:commandbutton></apex:form>
   
  
  <style type="text/css">
          .customPopup{
              background-color: white;
              border-width: 2px;
              border-style: solid;
              z-index: 9999;
              left: 50%;
              padding:10px;
              position: absolute;        
              width: 300px;
              margin-left: -250px;
              top:100px;
          }
          .popupBackground{
              background-color:black;
              opacity: 0.20;
              filter: alpha(opacity = 20);
              position: absolute;
              width: 100%;
              height: 100%;
              top: 0;
              left: 0;
              z-index: 9998;
          }
      </style>
  
  </apex:page>
  

• Now, to view the created visualforce page, you’ll need to go up to the browser address bar and navigate to the apex directory in your Salesforce.com account and type in the page name.
  For me, that is the following: https://na3.salesforce.com/apex/TestPopupPage
  or, you can create a new tab for this visualforce page, and can view it on click of the new created tab.

Here is the output comes up:

Created Visualforce Page:

After click on the button, modal popup is displayed:

To create a new VisualForce page go to “Your Name” > Setup > Develop > Pages and click the New button.

• Your Name” > Setup
  
• Develop > Pages
  
• New Page Button
  

You will now see the VisualForce page creation wizard. The very first thing you will want to do is give your new VisualForce page a Label. A label is a more esthetic name of what your page name will actually be.

Once you’ve typed your page label press the Tab key to set your cursor to the Name field. You will notice that the Name automatically took the value of “Test Page” from Label. SalesForce won’t allow you to have spaces in the name so replace your space with an Underscore character as shown below.

SalesForce will automatically give you some default VisualForce markup to work with as shown below.

<apex:page>
  <!-- Begin Default Content REMOVE THIS -->
 <h1>Congratulations</h1>
  This is your new Page
  <!-- End Default Content REMOVE THIS -->
</apex:page>

• And now save your page.
• For preview click on preview button.