How to manually clear EM Grid Control alerts

We can use the procedure outlined below to clear outstanding warning and critical alerts which are displayed on the EM Grid console home page. Normally these alerts get cleared automatically when the event ot threshold which caused these events in the first place is longer in the ‘critical’ or ‘warning’ state.

But there are some times when we wish to manually clear the alerts – there are some notes on Metalink which document cases where alerts are not cleared even after the event which caused them no longer exists – and also we may want to clear all the warning alerts which concern alerts related to logins by SYS user – out of the box auditing of the SYS user logins is enabled and every time SYS connects, it will generate a warning alert.

In a forthcoming post I will detail now to create a customised monitoring template and deploy it enterprise wide.

Let us see an example where we would like to clear outstanding alerts related to database ADRP.

The table SYSMAN. MGMT_TARGETS has information on the monitored targets and the table SYSMAN. MGMT_CURRENT_SEVERITY contains data related to all critical and warning alerts.

We can query the MGMT_CURRENT_SEVERITY table and see that there are two rows in the table pertaining to ADRP database.

SQL> select TARGET_NAME,TARGET_TYPE ,TARGET_GUID  from  mgmt_targets
  2  where TARGET_NAME like 'adrp%';

TARGET_NAME
--------------------------------------------------------------------------------
TARGET_TYPE
----------------------------------------------------------------
TARGET_GUID
--------------------------------
adrp
oracle_database
90BDD386E4F020F5356060DB6B94CAA3

SQL> select count(*) from mgmt_current_severity where TARGET_GUID='90BDD386E4F020F5356060DB6B94CAA3';

  COUNT(*)
----------
         2

We can use this procedure to delete the required rows – note that this block will in turn call the EM_SEVERITY package which will perform the clean up of dependant internal tables

DECLARE
   CURSOR c1 IS
     SELECT s.target_guid,
            s.metric_guid,
            s.key_value
       FROM mgmt_targets t JOIN mgmt_current_severity s
            ON s.target_guid = t.target_guid
      WHERE LOWER(t.target_name) LIKE 'adrp%'
        AND t.target_type = 'host';

BEGIN
    FOR r IN c1 LOOP
       em_severity.delete_current_severity(r.target_guid,
          r.metric_guid,
          r.key_value);
       DELETE from sysman.mgmt_severity
       WHERE
         target_guid = r.target_guid AND
         metric_guid = r.metric_guid AND
         key_value = r.key_value;

    END LOOP;
    COMMIT;
end;
/

SQL> select count(*) from mgmt_current_severity where TARGET_GUID='90BDD386E4F020F5356060DB6B94CAA3';

  COUNT(*)
----------
         0

10g EM Grid Control Notification Rules and Corrective Action Jobs

In 10g Enterprise Manager Grid Control, we can create corrective action or ‘fixit’ jobs which can automatically run if say a metric threshold is crossed or a particular event occurs. For example, we may have situations where we monitor the disk space used by the archive log files and as soon as the disk space free or available comes below a threshold, we would like another action to automatically occur which in this case could be to backup the archive log files to tape and then delete them to free up the disk space.

Read more about this feature in this worked example …….

Troubleshooting Agent Startup and Upload

If the agent in GRID control in not uploading to the OMS -

cd $AGENT_HOME/bin

oracle(DATABASE)@prdu030:./emctl start agent
Oracle Enterprise Manager 10g Release 4 Grid Control 10.2.0.4.0.
Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.
Starting agent ………. started.

oracle(DATABASE)@prdu030:./emctl status agent
Oracle Enterprise Manager 10g Release 4 Grid Control 10.2.0.4.0.
Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.
—————————————————————
Agent Version : 10.2.0.4.0
OMS Version : 10.2.0.4.0
Protocol Version : 10.2.0.4.0
Agent Home : /u01/oracle/agent10g
Agent binaries : /u01/oracle/agent10g
Agent Process ID : 856310
Parent Process ID : 897132
Agent URL : https://prdu030.bankwest.com:3872/emd/main/
Repository URL : https://prdu010.bankwest.com:1159/em/upload
Started at : 2009-08-07 12:27:29
Started by user : oracle
Last Reload : 2009-08-07 12:27:29
Last successful upload : (none)
Last attempted upload : (none)
Total Megabytes of XML files uploaded so far : 0.00
Number of XML files pending upload : 5004
Size of XML files pending upload(MB) : 20.53
Available disk space on upload filesystem : 69.35%
Collection Status : Disabled by Upload Manager
Last successful heartbeat to OMS : 2009-08-07 12:27:39
—————————————————————
Agent is Running and Ready

Steps to fix this issue -

cd $AGENT_HOME/sysman/emd

rm lastupld.xml agntstmp.txt

cd $AGENT_HOME/sysman/emd/upload

rm *.*

cd $AGENT_HOME/sysman/emd/state

rm *.*

No need to remove storage directory under $AGENT_HOME/sysman/emd/state

cd $AGENT_HOME/bin

./emctl start agent

./emctl status agent

If the Last successful heartbeat to OMS is not shown as successful, then secure the agent .

cd $AGENT_HOME/bin

./emctl secure agent

Make sure the password prompted for matches the OMS password – usually the sysman password for the OMS URL -

Once agent is secured -

oracle(DATABASE)@prdu030:./emctl status agent
Oracle Enterprise Manager 10g Release 4 Grid Control 10.2.0.4.0.
Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.
—————————————————————
Agent is Not Running

oracle(DATABASE)@prdu030:./emctl start agent
Oracle Enterprise Manager 10g Release 4 Grid Control 10.2.0.4.0.
Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.
Starting agent ……… started.

oracle(DATABASE)@prdu030:./emctl status agent
Oracle Enterprise Manager 10g Release 4 Grid Control 10.2.0.4.0.
Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.
—————————————————————
Agent Version : 10.2.0.4.0
OMS Version : 10.2.0.4.0
Protocol Version : 10.2.0.4.0
Agent Home : /u01/oracle/agent10g
Agent binaries : /u01/oracle/agent10g
Agent Process ID : 217220
Parent Process ID : 864280
Agent URL : https://prdu030.bankwest.com:3872/emd/main/
Repository URL : https://prdu010.bankwest.com:1159/em/upload
Started at : 2009-08-07 12:55:43
Started by user : oracle
Last Reload : 2009-08-07 12:55:43
Last successful upload : 2009-08-07 12:56:00
Total Megabytes of XML files uploaded so far : 2.00
Number of XML files pending upload : 6
Size of XML files pending upload(MB) : 3.96
Available disk space on upload filesystem : 69.31%
Last successful heartbeat to OMS : 2009-08-07 12:55:52—————————————————————
Agent is Running and Ready
oracle(DATABASE)@prdu030:

Troubleshooting Grid Control Agent issues with startup

Sometimes the GRID CONTROL agent will not start because an old HTTP process is still running on the host – this usually happens if the agent has crashed for any reason.

Example -

Check status of agent.

oracle(DATABASE)@hostname:./emctl status agent
Oracle Enterprise Manager 10g Release 4 Grid Control 10.2.0.4.0.
Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.
—————————————————————
Agent is Not Running

START AGENT -

oracle(DATABASE)@fhostname:./emctl start agent
Oracle Enterprise Manager 10g Release 4 Grid Control 10.2.0.4.0.
Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.
Starting agent …… failed.
Failed to start HTTP listener.
Consult the log files in: /u01/oracle/agent10g/sysman/log

To solve the problem.

CHECK IF PORT 3872 – PORT USED BY AGENT IS IN USE.

oracle(DATABASE)@hostname:netstat -an | grep 3872
tcp4 0 0 *.3872 *.* LISTEN

oracle(DATABASE)@hostname:ps -ef | grep emagent
oracle 864486 1 0 Jan 27 – 5:36 /u01/oracle/agent10g/perl/bin/perl /u01/oracle/agent10g/bin/emwd.pl agent /u01/oracle/agent10g/sysman/log/emagent.nohup
oracle 1593344 864486 0 Jan 27 – 97:38 /u01/oracle/agent10g/bin/emagent

KILL ALL EMAGENT PROCESSES STILL RUNNING -

oracle(DATABASE)@hostname:ps -ef | grep emagent | awk ‘ {print $2}’ | xargs kill -9

oracle(DATABASE)@hostname:ps -ef | grep emagent| grep -v grep

No emagent process running now.

cd $AGENT_HOME/bin

oracle(DATABASE)@hostname:pwd
/u01/oracle/agent10g/bin

START AGENT

oracle(DATABASE)@hostname:./emctl start agent
Oracle Enterprise Manager 10g Release 4 Grid Control 10.2.0.4.0.
Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.

AGENT WILL START.

EM Grid Control target discovery after a clone

If a database is cloned or recreated on another machine with the same name, the new database will not be discovered on the GRID.

There is a reason behind this .

Example

Database called test already discovered on GRID – database on ABC machine.

Create or clone a database with the same name test on another machine XYZ.

Try to discover this new database on the GRID using the manual option or the option to monitor via grid control while creating the database.

The new database test will not be discovered on the GRID and cannot be seen in the list of targets on the GRID .

That is because a databse with the same name the test database has also has been discovered and runs from another box.

Only when a manual install was attempted and the name of the instance was issued as test.domain instead of just test that it worked .

As you can see below – the repository ( sysman in emrep database ) has a table called mgmt_targets which maintains these details and does not allow duplicates( so a little tweaking is necessary )

The agent tried to upload test automatically when installed on the new box and start it up- and it failed to do so as this entry exists in the table.

Currently both databases are disovered – run this command in the emrep ( repository database) as sysman.

SQL> select target_name, from mgmt_targets where target_name like ‘%TEST%’;

TARGET_NAME
test
test.domain

EM Grid Control Agent Target Discovery

Sometimes , you may install the 10g OEM GRID agent before you actually create a new database or you may add a new database to an existing box.

This new database will have to be discovered by GRID CONTROL.

I created a database called test on a box which already had a Grid agent running on it.

This database has to be discovered by the agent and it’s details uploaded to the GRID.

Make sure the ORAINVENTORY location in /etc/oraInst.loc matches the location when you installed the AGENT.

Go to the new AGENT_HOME/bin and issue command agentca- d

test:/u01/ofsap/agent10g/bin> ./agentca -d
Stopping the agent using /u01/ofsap/agent10g/bin/emctl stop agent
Oracle Enterprise Manager 10g Release 4 Grid Control 10.2.0.4.0.
Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.
Stopping agent … stopped.
Running agentca using /u01/ofsap/agent10g/oui/bin/runConfig.sh
ORACLE_HOME=/u01/ofsap/agent10g ACTION=Configure MODE=Perform
RESPONSE_FILE=/u01/ofsap/agent10g/response_file RERUN=TRUE
INV_PTR_LOC=/etc/oraInst.loc
COMPONENT_XML={oracle.sysman.top.agent.10_2_0_1_0.xml}
Perform – mode finished for action: Configure
Perform – mode finished for action: Configure

You can see the log file:
/u01/ofsap/agent10g/cfgtoollogs/oui/configActions2009-06-18_01-13-37-PM.log
test:/u01/ofsap/agent10g/bin>

Output of log -
/u01/ofsap/agent10g/cfgtoollogs/oui/configActions2009-06-18_01-13-37-PM.log

The action configuration is performing
——————————————————
The plug-in Agent Configuration Assistant is running

Performing free port detection on host=prdu024.bankwest.com
Performing targets discovery and agent configuration
Starting the agent
AgentPlugIn:agent configuration finished with status = true

The plug-in Agent Configuration Assistant has successfully been performed
——————————————————
The action configuration has successfully completed
###################################################

Now login to the database as sys, unlock the dbsnmp user and also change
the password of dbsnmp user – for example to temp.

Login to the GRID , and click on targets – choose the new box and then the
new database and configure the dbsmp user .

Once this is complete, the GRID screen will show both the agent and
database on the targets page.

Grid control- Manual deletion of jobs

MANUAL DELETION OF GRID CONTROL JOBS.

Sometimes GRID CONTROL jobs may show as RUNNING although they have completed.

This will restrain the job from returning to the ‘SCHEDULED’ state and thereby will not run.

To manually remove a job in the job queue from the backend -

Login to the repository owner in the Grid Control repository database -

Repository owner – SYSMAN

Repository database – EMREP

SYSMAN@emrep> select job_id, job_name, job_owner from mgmt_job where job_name like ‘%BACKUP%’;

JOB_ID
——————————–
JOB_NAME
—————————————————————-
JOB_OWNER
——————————————————————————–
5C153E5F858740BCE0430AFEC84040BC
TEST BACKUP

Firstly stop all executions of this job.

SYSMAN@emrep> exec mgmt_job_engine.stop_all_executions_with_id(‘5C153E5F858740BCE0430AFEC84040BC’,TRUE);

PL/SQL procedure successfully completed.

SYSMAN@emrep> commit;

Commit complete.

Finally remove this job from the job_management engine.

SYSMAN@emrep> exec mgmt_job_engine.delete_job(‘5C153E5F858740BCE0430AFEC84040BC’);

PL/SQL procedure successfully completed.

SYSMAN@emrep> commit;

Commit complete.

EM Grid Agent targets.xml Modification After Database Upgrade

AGENT RECONFIGURE AFTER DB UPGRADE.

If you upgrade an Oracle database or start the database from a new Oracle Home, the following needs to be done for GC to recognise the change and for the database to be monitored via the GRID.

Shutdown the agent on the box -

cd AGENT_HOME/bin

./emctl stop agent.

Next make a copy and then edit the following file -

Location – AGENT_HOME/sysman/emd/targets.xml

Targets AGENT_TOKEN=”1982fef1f5fe5b9c64f3d1e2edd8c4e875c889df”
Target TYPE=”oracle_emd” NAME=”frmbwds06:3872″/
Target TYPE=”host” NAME=”frmbwds06″/
Target TYPE=”oracle_listener” NAME=”LISTENER_frmbwds06″
Property NAME=”ListenerOraDir” VALUE=”/u01/oracle/10.2/network/admin”/
Property NAME=”LsnrName” VALUE=”LISTENER”/
Property NAME=”Machine” VALUE=”10.254.200.234″/
Property NAME=”OracleHome” VALUE=”/u01/oracle/10.2″/
Property NAME=”Port” VALUE=”1521″/
/Target
Target TYPE=”oracle_database” NAME=”adra.bankwest.com”
Property NAME=”OracleHome” VALUE=”/u01/oracle/10.2“/
Property NAME=”UserName” VALUE=”dbsnmp”/
Property NAME=”MachineName” VALUE=”10.254.200.234″/
Property NAME=”Port” VALUE=”1521″/
Property NAME=”SID” VALUE=”adra”/
Property NAME=”ServiceName” VALUE=”adra.bankwest.com”/
Property NAME=”password” VALUE=”8b139f39a2544353″ ENCRYPTED=”TRUE”/
Property NAME=”Role” VALUE=”NORMAL”/
/Target
/Targets

Make changes to ORACLE_HOME – Notice above I have changed the OH for adra to point to the 10g HOME.

Restart the agent -

cd $AGENT_HOME/bin

./emctl start agent -

After a minute or so -

./emctl status agent – to make sure uploads are happening to the repository on prdu010.

oracle@frmbwds06 bin ./emctl status agent
Oracle Enterprise Manager 10g Release 4 Grid Control 10.2.0.4.0.
Copyright (c) 1996, 2007 Oracle Corporation. All rights reserved.
—————————————————————
Agent Version : 10.2.0.4.0
OMS Version : 10.2.0.4.0
Protocol Version : 10.2.0.4.0
Agent Home : /u01/oracle/agent10g
Agent binaries : /u01/oracle/agent10g
Agent Process ID : 1593568
Parent Process ID : 2007106
Agent URL : https://frmbwds06:3872/emd/main/
Repository URL : https://prdu010.bankwest.com:1159/em/upload
Started at : 2009-03-14 13:20:18
Started by user : unknown
Last Reload : 2009-03-14 13:20:18
Last successful upload : 2009-03-17 09:33:06
Last attempted upload : 2009-03-17 09:37:39
Total Megabytes of XML files uploaded so far : 102.26
Number of XML files pending upload : 0
Size of XML files pending upload(MB) : 0.00
Available disk space on upload filesystem : 30.16%
Last attempted heartbeat to OMS : 2009-03-17 10:43:55
Last successful heartbeat to OMS : 2009-03-17 10:12:39
—————————————————————
Agent is Running and Ready
oracle@frmbwds06 bin

10g EM Grid Control Installation

The document below gives a clear idea of how to install 10g EM grid control and upgrade the environment to 10.2.0.3

There are 3 components in a grid control installation.

1. Oracle Management Server – OMS
2. Creation of Grid control repository database.
3. Oracle grid control agent

View the note to read a step by step guide to the installation.

Download …

10g EM Grid Control Agent Push Deployment

It is possible to deploy the Grid Control OEM agent to various machines you want monitored via the push method. This method installs the agent on other machines without the need to copy the software to those machines.

This is very useful in case there are many machines to be monitored by Grid Control.

Download …