What is special about this place?

Here is a trivia question for you, what is special about this building?

If you think you know what significant (CICS related) event went on in this building, add a comment. The first one to guess correctly wins … a “I ♥ CICS” sticker.

CICSPlex SM traces

To an experienced CICS user, CICSPlex SM traces can seem weird compared to ‘vanilla’ CICS traces, so I am going to try and explain what is going on with them.

First a little bit of history

When CICSPlex SM was first introduced it was a separately orderable product – it was not part of the CICS TS package (the CICS TS package did not exist!). Because of this, the CICSPlex SM developers had to play by the rules (to keep the IBM lawyers happy), in that it had to use only published CICS interfaces, e.g. using the standard CICS APIs/SPIs and only programming interfaces that could be determined from the publicly available CICS documentation.

As the interface into CICS tracing is EXEC CICS ENTER TRACENUM, that is what CICSPlex SM uses to write trace records. This is also why CICSPlex SM trace records appear to CICS as so called ‘user’ trace records.

Nordic Guide

If you are attending the GSE Nordic event in Elsinore, Denmark, next week, I’d be very pleased to met you in person and have a chat about CICS, CICSPlex SM, System z, debugging and support, or anything else that you wanted to talk about. I am giving three presentations – on CICS/VSAM performance, CICS trace and CICSPlex SM, and the latest news in CICS monitoring and statistics. Please come round and say hello if you would like to…  best wishes – Andy

CSFE storage violation traps

Chris’s post about SLIP made me think about CICS storage checking and CSFE. As this blog is supposed to be for people with all levels of CICS skills, I thought I’d try and write something for beginners about CICS storage checking.

Note for Chris: Yes, it is called “storage” not “memory”! 😉

For CICS task lifetime related storage, CICS puts some eyecatchers on the start and end of the buffer that it obtains for you. These are called storage crumple zones. If you look at some CICS storage using a tool like CEDF, you can see these at the bounds of the area obtained for your user storage. For example:

15014A80   000010   E4F0F0F1 F6F8F0F0 838983A2 979385A7   U0016800cicsplex 
15014A90   000020   00000000 00000000 E4F0F0F1 F6F8F0F0   ........U0016800 

In the above example U0016800 is the crumple zone. The U means this is user key storage, (you can also get a C for CICS key storage). The number after the U is the task number that owns the storage. Task number 16800 in this example. Notice that there are two occurrances of the U0016800 string – a leading and a trailing crumple zone. In this example I used EXEC CICS GETMAIN to obtain 8 bytes of storage (CICS rounded it up to 16).

When CICS obtains storage for your task – either directly because you’ve issued an EXEC CICS GETMAIN or indirectly e.g. task working storage – then CICS will setup the storage crumple zones.

When the storage is freed – again either directly because you’ve issued an EXEC CICS FREEMAIN or indirectly e.g. when your task ends, then CICS checks the crumple zones associated with the storage. If the crumple zones are not correct, then a storage violation is detected.

So if my program wrote more than 16 bytes worth of data to the piece of storage above, the trailing crumple zone would be changed. However the overwrite could occur in my program at any time, yet the error might only be detected when the task ends.

When CICS does detect the storage violation, the CICS sysprog, will then typically need to try and figure out which CICS application programmer to harass. However in order to do this, the sysprog needs some evidence as to which program is likely to be at fault. The task that suffered the violation might be the victim of another tasks errant program. If the task was long running, and the storage violation was detected at end of task, then there may be a significant amount of code or other tasks that could have run.

This is where the CSFE transaction may be able to help. For reproducable storage violations – typically on a test system – you can use the storage violation trap function of the CSFE.

Using CSFE DEBUG,CHKSTSK=CURRENT will cause similar storage checking that occurs when user storage is freed to occur after every CICS trace record is produced. So as long as your CICS is running with CICS internal trace active and the tasks are invoking CICS functions that cause trace enties to be written, then CSFE has something to work with.

When the storage violation trap is active, then when the CICS trace program writes a trace record to the internal trace table, the CICS storage manager domain will be called to check that the storage crumple zones of the current task are intact. If they are, processing will continue as normal. However if not, the storage violation trap will be tripped, and you will then get a dump.

Then armed with the handy dump and IPCS (or your other favourite dump debugging aid) you can then examine the trace to determine which programs were in control and what commands they were issuing. This should hopefully help narrow down the culprit and identify which program, and possibly even sections of code in the program deserve a close inspection.

There are other CICS facilities e.g. storage protection, transaction isolation, re-entrant program protection and EXEC command storage checking that can help with resolving storage overlays in CICS, so all is not lost if CSFE doesn’t deliver the goods. However when you do have a situation where it can be used, it can be invaluable.

Technical authorship

An area that I have been very active in over the past 15 years or so has been technical authorship. I’ve done a lot of writing about CICS, transaction processing, debugging, and problem determination. I’ve found that this has been an effective way for me to communicate with CICS customers, and with other IBMers too. It has also helped me enormously because I have had to learn about an area before I can go ahead and write about it myself, so the act of writing has forced me to try and master the subject matter beforehand 🙂 As well as working with the IBM ITSO team on redbooks and education classes, I have also been published in a number of external journals and publications. I’d be interested in knowing what experiences other authors have had in writing and publishing about CICS, CICSPlex SM, transaction processing or IBM System z in general. I know my fellow blogger Phyllis has been very active with her writing over the years… what about anyone else out there?

Running CICS?

OK, not Running CICS, really, but CICS runners. On Saturday the 17th of May, two CICS managers (one development, one test), a CICS developer and two CICS testers will be running 50 miles in support of MacMillan Cancer Support. For details see here.

Here are a few facts about the CICS runners:

• One of them will be spending the entire time trying to figure out how the GPS on their phone works.
• One will be quietly sobbing to themselves whenever they have to run past a pub without actually going in.
• One has promised us that they won’t be wearing shorts that are too tight.
• One we are secretly hoping does wear tight shorts.
• One will disappoint us if they do actually manage to wear socks that match

Seriously though folks, good luck for the run!

CICS TS in your SOA – Issue 3

An excellent resource with links about CICS in the shape of redbooks, external articles, reports, whitepapers, brochures, podcasts, datasheets and more! To quote a slide from the presentation:

“This presentation provides a ‘portal’ to more information on the CICS ecosystem. All the items in this presentation have hyperlinks that are active in ‘Slide Show’ mode. This will be an evolving document, with refreshes distributed every few months. To receive these refreshes, please just email : batesan@cn.ibm.com“

Get the presentation here: cics-ts-in-your-soa-links-issue-3.ppt

Update: Fixed some bad grammar.

IBM CICS Early Programs

If you had a chance to go to Impact 2008, you might have been given one of these cards:

If you didn’t go to Impact 2008, and you think you are up to the challange of getting involved in CICS Early Program offerings, get in touch with the BetaWorks team at CICSEP@uk.ibm.com

Using MVS SLIP to detect memory changes

Every now and then you come across a problem that you just can’t fix with the tools in your toolbelt. The problem? How to detect what part of your program is changing a specific area of storage? The program in question being CICSPlex SM and the storage a 1 byte flag stored in a MVS dataspace. A code search only proved that the flag was not being modified anywhere obvious and a trace of the suspect modules proved unhelpful.

So failing to batter the problem into submission with the normal tools, a kind colleague pointed me at the MVS SLIP (Serviceability Level Indication Processing) operator command. A few painful moments passed as the correct command incantation was established and lo and behold I received a dump with the PSW pointing handily at the module that changed the storage location. Thankfully we use eye-catchers at the top of each internal method so detecting the module is simply a matter of browsing the dump at the address suggested and scrolling up.

Generally within the CICS TS team the most commonly used tools for debugging problems are CEMT, CECI, IPCS, AUX trace and an internal interactive debugging tool called Source Language Debug (SLD). In CICSPlex SM development/service we also use the customer-provided COD0 transaction to quickly walk-around the system control blocks and dump out specific data.

The SLIP command is definitely one tool I’ll be adding to my toolbelt. What tools can you not live without?

Sharing CICSPlex SM WUI data repositories

The CICSPlex SM Web User Interface (WUI) requires a repository that is used to store CICSPlex SM view sets, menus, user group profiles, user favourites (I am in the UK so hopefully I can get away with the proper spelling) etc. As CICS is great at dealing with files, a KSDS called EYUWREP was chosen as this repository.

CICS has a number of different facilities that let you share files. For example function shipping, shared data tables and RLS. However a slight snag in the design of the WUI object service meant that it was not possible to use any of these facilities and maintain integrity of the EYUWREP file.

In a nutshell, the WUI object service may use multiple physical records in the EYUWREP file to represent one logical record, for example a view set may contain any number of views. It is the view set that is the ‘container’ for the views, so is saved and loaded as a single entity.

The WUI does have a locking service, but this were built on the assumption that each WUI server had its own unique EYUWREP file. So the locking service only provides for locks within a single WUI server. This is one of the reasons why it is documented that the EYUWREP file can NOT be shared between WUI servers.

Prior to CICSPlex SM V3R2M0, there were no checks on the EYUWREP file definition to ensure that it had not been changed to RLS or a remote file or a data table. We didn’t expect customers to change the IBM supplied definition, but some of them did anyway.

When these customers (with ‘modified’ EYUWREP file definitions) migrate to CICSPlex SM V3R2M0, then the WUI server will fail to initialise with message:

EYUVS0916E FILE EYUWREP has invalid value: YES for option: RLSACCESS

This message is issued in V3R2M0 because we made some changes to the way CICSPlex SM resource definitions are built. Bootstrap definitions are created during CICS startup, and then if the bootstraps are invoked, the remaining definitions are created on the fly. Because we acknowledged that some people change definitions, we allow user definitions, but we now check the definitions to make sure it is something vaguely close to the IBM supplied one. If not, the EYUVS0916E message is issued.

So even though you might have a ‘modified’ EYUWREP definition and the modification was for something that was documented as not being supported due to the risk of corrupting data, a change has recently been made to CICSPlex SM V3R2M0 by APAR PK61740.

The PTF for this APAR changes the WUI locking service so that it can provide protection for the logical EYUWREP records. This allows the EYUWREP to be shared within a sysplex between multiple WUI servers via RLS. Note that that we are only supporting RLS for the sharing, not function shipping, or shared data tables etc, just RLS.

So take a look at this APAR if you want to share WUI server data repositories. Please make sure you check out the hold info in the PTF cover letter for further details