You probably had the AdBlock for Safari extension and installed the new AdBlock for Safari app. You should use either the app or the legacy extension, not both.
If you choose to use the app, disable "AdBlock 2.67.0 by BetaFish" in Safari Extensions Preferences. (The version number may be different from "2.67.0," but it should be higher than the AdBlock app version.)
If you choose to use the extension for now, disable "AdBlock Safari Icon 1.3.1 from AdBlock" and "AdBlock 1.3.1 from AdBlock" instead. (Again, the version number may be different, but it will be lower than the "AdBlock by BetaFish" extension version.)
How do I know which AdBlock to keep?
The app doesn't yet have all the features that are available in the extension. We'll be adding more features over the next few months. If a missing feature is important to you, we recommend you keep using the extension until the feature becomes available in the app.