To edit, pause or delete an alert, you need to go to the Smart Alerts section.
For each alert set, 3 actions can be performed: Edit, Pause/Play, Delete.
Click Delete if you want to delete an alert that you have. Deleting the alert means no more mentions will be sent in the future for that alert.
Click Pause if you want to temporary disable an alert. While the alert is paused, no alert notifications will be sent for it. If you want to enable it afterwards, click on Play.
Click Edit if you want to make changes to a current alert. Once you click here, a pop-up similar with the one below will appear, where you'll be able to perform the changes you need.