Stephen Weblog

Identify the JVM process and attach truss to it -

`truss -o /tmp/truss.log -f -e -p <JVM pid>`

In AIX, shared objects (*.so) and shared libraries (*.a) have a specific behavior based on permission assigned to them, in particular - read permission for others - which would mean ( chmod o+r <shared object file> ).
If read permissions for others are given, the shared object gets loaded into the system global shared object segment’, which the executables will benefit from the fact that the shared objects are already loaded and don’t have to be called again. These shared objects will continue to remain in memory even after the program exits, making all future runs of the program faster, by reusing the already loaded shared objects. On the other hand, if the ‘read permissions for others’ are removed (o-r), the ’shared object’s text and data’ will be loaded into the ‘process private segment’ rather than the system shared object segment. Which means every time a program using shared object is called, it maintains a private copy of the shared object’s text and data introducing 2 kinds of bottlenecks. The first one being - if a program is called twice, 2 private copies of the text and data get loaded. The second one - private copies of text and data don’t remain in memory after the program exits (basically acting like non-shared objects), so there are expensive memory calls happening to the shared objects every time a program is called…

On many occasions you could have come across this error, mmap() failed to load some library

” Call to mmap() failed - TEXT /apps/rup4tst/apps/tech_st/10.1.2/lib32/libzrc.sl
/usr/lib/dld.sl: Not enough space”

These issues are specific to operating system, mmap function call could fail due to various reasons.

• Shared memory allocation
• Parameters values for maxdsize, maxtsize, maxssize etc

You will have to check the Kernel parameters in the machine, as well as you will have to check the shared memory allocation. You will have to review the allocated shared memory in the system. If there are many chunks of memory occupied, I would suggest to reboot the machine and try running the failing process… (Believe me reboot has resolved many such problems for me)

Shared Memory is one of the beautiful concepts in Unix OS, these materials were taken from here.

Shared memory (SHM) - is another method of interprocess communication (IPC) whereby 2 or more processes share a single chunk of memory to communicate. The shared memory system can also be used to set permissions on memory, allowing for things like malloc debuggers to be written.

Types of Shared Memory available in most Unix flavours

• BSD mmap
• System V IPC

Key points to be noted down in shared memory implementation

• Uniquely naming the segment
• Specifying access permission
• Race condition

BSD mmap - Three important concepts on BSD (Berkeley Software Distribution) implementation

• Mapping files
• Permission and access
• Synchronization

Mapping files - When you request a shared segment under the BSD implementation, what you are really doing is requesting that a file (actually, a file descriptor ‘fd’, which was already opened by open(2)) of length ‘length’ be mapped into the address pointed to by ’start’ of your proceess space starting ‘offset’ bytes into the file. If start is NULL, the address is determined by the kernel. NOTE: You must know the length of the file being mapped (see fstat(2)), and the length of your segment will not change from this initial length. This means if write past the end of the segment, the file will not be appened to. You will get a segmentation fault instead.

The kernel then instructs the MMU of the CPU to generate a “page fault” when a section of this region of memory is accessed. When the kernel gets this page fault, it gets a page (4K) of data from the corresponding location in the file, and reads it into memory. The kernel only does this once per page of data in your mapped segment. This also means that no physical memory is allocated untill you start accessing the pages of the mapped region. The one exception to this is when mapping /dev/zero. Maps to /dev/zero are all prefaulted (zeroed), and the memory thus actually allocated.

Permission and access - The standard file modes are checked by the kernel to determine whether or not a process has access to map a file (shared or not). In addition, you may also specify permissions on the segment itself. These permissions describe to the kernel what may be done to the actual memory (Can it be executed? Can it be read? Written to?) The mprotect system call can be used to change permissions on any section of previously mapped memory that starts on a page boundary.

When an attempted mmap to a file fails, mmap returns -1 (not NULL!), and errno is set. However, when you try to access a segment in a way that violates its permissions, a segmentation fault is generated.

Synchronization - When you write to a your shared segment, your changes do not appear immediately to the file until you perform an msync(2). They are, of course, visible to other processes who also are sharing your segment. Additionally, you may make the segment private, which means that if another process maps the file, it does not see changes to your version of the mapped file, and vice versa. Another characteristic that might be confused with a private mapping is an anonymous mapping. Anonymous mappings are not associated with files at all, and hence are not shared memory. We will use them to implement our malloc debugger however.

System V IPC - System V IPC is meant to provide and entire IPC mechanism. As such, it is more heavyweight than BSD mmap, and provides three methods of communication

• Message queues
• Semaphores
• Shared segments

Message queues -
Message queues provide a memory based FIFO between two processes. The primary difference between a System V message queue and a socket or named pipe is that message queues may have multiple processes reading and writing from and to them, or no readers at all.

#include <sys/types.h>
#include <sys/ipc.h>
#include <sys/msg.h>

int msgget(key_t key, int msgflg);
int msgsnd(int msgid, struct
    msgbuf *msgp, int msgsz,int msgflg);

int msgrcv(int msgid, struct msgbuf
*msgp,int  msgsz,long msgtyp,int msgflg);

struct msgbuf {
    long mtype;     /* must be > 0 */
    char mtext[msgsz]; /* message data */
};

Much like for shared memory, you must create a message queue using the msgget(2) system call. It takes a key and a flag paramater and returns an integer id for your segment. The flag can be combinations of IPC_CREAT and IPC_EXCL, as well as the permissions bits on the segment as described above. The System V IPC provides two symmetric system calls for sending and receiving messages: msgsnd and msgrcv respectively.

Semaphores -

Semaphores are data structures that are used for synchronization between two or more processes. Basically they can be viewed as a single integer that represents the amout of resources avaialable. When a process wants a resource, it checks the value of the semaphore, and if it it nonzero, it decrements the appropriate number from the semaphore in accordance to the amout of resources it wishes to use. The kernel will block the process if the semaphore is zero or doesn’t have a value high enough for the decrement. In System V IPC, you may request a set or an array of semaphores at once. To create a set of semaphores, you must use the semget(2) system call. It takes a key, a number of semaphores to create, and a flag paramater and returns an integer id for your segment. The flag can be combinations of IPC_CREAT and IPC_EXCL, as well as the permissions bits on the segment as described above.

Shared Segments -

Shared segments in System V are a bit different than those obtained with BSD’s mmap. First of all, while you do need a key for a segment which is usually obtained by running ftok(2) on some file in the filesyetem, System V shared segments exist only in memory. Because of this, System V shared segments must persist after the termination of a process. It is the responsibility of the programmer to ensure that all of his segments are removed from the system, otherwise his memory leaks become system wide.

To create a shared memory segment, you must use the shmget(2) system call. It takes a key, a size, and a flag paramater and returns an integer id for your segment. The flag can be combinations of IPC_CREAT and IPC_EXCL, as well as the permissions bits on the segment as described above.

Once the segment is created and you have it’s id, you must attach it, or map it into memory. This is done with shmat(2). shmat works much the same as mmap in that you can specify an address to map to, but the address must be a multiple of SHMLBA, unless you specify SHM_RND in the flags paramater.

Various operations can be performed using shmctl(2) IPC_STAT can be used to fill in the struct shmid_ds structure. You can then set various fields (including changing ownership and premissions of ipc_perm) using IPC_SET. But perhaps the most important ctl operation is IPC_RMID. When all of your programs are done using a segment, you MUST shmctl it with IPC_RMID (the shmid_ds structure may be NULL), or else the segment will remain in memory forever.